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Foreword by Jill Journeaux 


Charcoal, the interaction of fire and carbon, and naturally occurring pigments such 
as red clay were the first materials used by homo sapiens to record their presence in 
the world. Throughout time, artists have been curious about tools and technologies 
that have emerged from adjacent or alternative domains and have been open to 
reapplying those tools and technologies into new areas in innovative ways. In recent 
times, it is computers and interactive systems that have driven this curiosity, but we 
should also remember the great variety of forms of technology that have expanded 
and accelerated art practice throughout history. These include photography, the use 
of hot metals such as bronze and steel to create form, the unique qualities of 
plastics, the simple processes of fire in raku ceramics and in making charcoal, and 
the creation of synthetic pigments in 1704. The application of these inventions and 
innovations to artmaking has involved highly skilled technologists who have 
worked in the spaces between art and science to manifest new possibilities of form 
and expression. 

In order to make sense of the world and our place within it, we attempt to 
construct conceptual models. Artists are often attracted to the apparent certainties of 
science but bring to it expressions of doubt and uncertainties about a rationalist 
approach. They utilize dialogues between interiority and exteriority in order to 
confirm the nature of being in the world as known or understood through inner 
emotional and spiritual worlds, and use artmaking as a process that searches for 
established and new truths based on knowledge and science but felt and intuited 
through visual recognition by others. They place value on the unknown as well as 
the known. 

In this book, Rae Earnshaw, Susan Liggett, Peter Excell and Daniel Thalmann, 
explore the spaces between technology, design and the arts and ask questions of the 
challenges and opportunities that are presented by interdisciplinary research across 
these fields. The inclusion of a series of case studies allows for consideration of the 
value of interdisciplinary research which incorporates practitioners, practice and 
practice research and enables explorations of the manner in which technological 
innovations shape cultural manifestations and extend the capacity for artists to 
articulate experiences of being, thinking and questioning. 
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viii Foreword by Jill Journeaux 


Rae Earnshaw and Susan Liggett first collaborated in 2015 when Liggett curated 
the exhibition Carbon Meets Silicon, in Oriel Sycharth. This exhibition included 
works by artists who collaborate with scientists, or technologists and was accom- 
panied by a symposium as part of the 6th International Conference on Internet 
Technologies and Applications. The conference brought together researchers and 
developers from academia and industry across engineering, computing and art and 
design, and was inspired by Alan Turing’s explorations of artificial intelligence as 
opposed to cognitive psychology. Artwork in the exhibition explored a range of 
media practices including film and video installation, painting, printmaking, jew- 
ellery and sound-based works. 

As part of the 7th International Conference on Internet Technologies held in 
2017, Earnshaw and Liggett organized the exhibition Carbon Meets Silicon II 
which brought together fifteen diverse artists using science and technology to better 
understand the world we live in. The thematic exhibition used carbon and silicon as 
metaphors for the evolving nature of art practice in a digital era. Carbon was used to 
refer the materiality of art objects and silicon to refer to how an artworks’ physical 
presence may be questioned by new media and technologies, thus bringing artist 
and scientist closer together. 

The editors of this book are concerned to reframe perceptions of dualism and 
difference between artists and scientists to ensure that they are not set apart but can 
instead find ways of working together to solve problems. However, if we consider 
the processes and ambitions associated with creativity supposed differences 
between artists and scientists may become less distinct. Creativity theorist Mihaly 
Csikszentmihalyi reminds us that: creativity does not happen inside people ’s heads, 
but in the interaction between a person’s thoughts and a sociocultural context. It is 
systemic rather than an individual phenomena (1997: 23). In his book, Creativity 
Csikszentmihalyi argues that levels of creativity do not just depend upon creative 
individuals but are also affected by the willingness of domains and fields to rec- 
ognize novel ideas and artefacts (1996: 31). 

Writing about successful collaborative partnerships in both science and the arts 
Vera John-Steiner emphasizes the value of complementarity stating: Each indi- 
vidual realizes only a subset of the human potential that can be achieved at a 
particular historical period (2000: 40). She argues that the modes of thought 
involved in generating explanatory principles are not necessarily the ones used in 
communicating scientific discoveries and refers to Reuben Hersh’s idea of the 
‘backstage’ of discovery, which may consist of fragments, the conversational, 
informal and intuitive propositions, and tentative notions. As with artists some 
scientists only share their complete and finished products, but others reveal their 
thought processes through notebooks, drawings, journals and other preparatory 
thinking processes and formats [3]. Technologists, artists and scientists visualize the 
as yet unmade and the possible as well as the actual and the clearly thought and 
understood. 

In an interview with Alfred Appel Jr., in 1966, Vladimir Nabokov reacted to 
C. P. Snow’s assertion that the gap between the two supposedly separate cultures of 
science and the arts was unbridgeable. He argued that science has an artistic and 
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creative side and that the arts require scientific truths, saying: I certainly welcome 
the free exchange of terminology between any branch of science and any raceme of 
art. There is no science without fancy, and no art without facts (1973: 78-79). 


Jill Journeaux 

Professor of Fine Art 

Centre for Arts, Memory and Communities 
Coventry University, UK 
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Foreword by Jon Peddie 


Technology, Design and the Arts—Opportunities and Challenges is a book for the 
times. Never before has there been such a confluence between technology and art as 
today. Amazing new developments in computer graphics, not the least of which is 
the democratization of the tools, has empowered and emboldened designers, and 
want-to-be artists all over the world. Art, from fine art to set backdrops, video 
games and graphics novels, has permeated our lives, imaginations and culture at all 
levels—and we are richer for it all. 

It all begins with the software tools, which as the book beautifully illustrates, 
have been used from the earliest times to create and modify artistic works. And the 
concept is not limited to just graphics art, but various types of tools for creating 
image and musical instruments—computer technology has been used to create 
digital images and audio. 

The authors are to be congratulated for being brave enough to examine and 
discuss the symbiotic relationship between artistic works and the cultural context in 
which they are produced. 

Computers have been used to mimic the way classical artists did things, their 
attempts were not always successful. The technology can provide continuity by 
taking traditional methods and techniques and making them more efficient and 
effective. However, it can also provide divergent results and open up new per- 
spectives and paradigms—sometimes intentionally, and sometimes accidentally. 
The authors argue that it can produce a greater understanding of artistic processes 
and how they are implemented in practice. The reader will be the judge. 

The authors believe a tighter coupling of the interplay between the goal of the 
creator, the selection and use of appropriate tools, and the materials and repre- 
sentations chosen has been created. They cite Gombrich who argued that, There 
really is no such thing as Art. There are only Artists. Thus, although creative works 
may generate a variety of interpretations in the minds of observers, say the authors, 
the focus also remains on the creator, and the vision and insight that causes them to 
create. Thus, what has been created provides insight into the cultural context and 
social environment at the time of creation. Powerful ideas are very effectively 
argued and demonstrated in the book. 
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xii Foreword by Jon Peddie 


The book proposes to define and detail the relationship between the artist and 
their works by the types of tools they use, and the environments that have facilitated 
and extended these processes. Such tools for graphics say the authors have 
encompassed computer technology, computer environments, and interactive devi- 
ces for a range of information sources and application domains. They have also 
provided new kinds of created works which are able to be viewed, explored, and 
interacted with, either as an installation or via a virtual environment such as the 
Internet. 

I agree, and think this introduces new dimensions of understanding and expe- 
rience for both artist and the public’s contact with, and hopefully understanding 
of the works that are produced. But it doesn’t come for free. A variety of inter- 
disciplinary opportunities and issue are raised, and the authors bravely examined 
them in detail. Spanning from the classics such as Leonardo da Vinci to contem- 
poraries like David Hockney the opportunities for artistic and creative expression 
have transformed the worlds of which they are an integral part. 

The contributions and themes in this book are rich in information in various 
forms. The interaction of technology and computer generation of information with 
artefact creation and dissemination is discussed and the reader is left with a sense of 
wonder, questions and hopefully a few answers. The topic is, and may always be, a 
work in process. That is reflected in that this book is a continuation of, and 
development from, the research and development detailed in the earlier book— 
Earnshaw R. A. (2017): Art, Design and Technology: Collaboration and 
Implementation. 


Dr. Jon Peddie 
Jon Peddie Research 
Tiburon, CA, USA 


Preface 


This volume explores emerging aspects of the relationship between artists (and 
other creatives) and their created works, and also how a variety of tools and 
environments have facilitated and extended these processes. Such tools encompass 
computer technology, computer environments, and interactive devices, for a range 
of information sources and application domains. They also provide new kinds of 
created works which are able to be viewed, explored and interacted with, either as 
an installation or via a virtual environment such as the Internet. This introduces new 
dimensions of understanding and experience for both artist and the public’s rela- 
tionships with the works that are produced. This has raised a variety of interdis- 
ciplinary opportunities and issues. From Leonardo da Vinci to David Hockney the 
opportunities for artistic and creative expression have transformed the worlds of 
information of which they are an integral part. 

Tools have been used from the earliest times to create and modify art works and 
sculptures. Naturally occurring pigments have been used for cave paintings, but 
these were more than pictorial representations: they were inextricably linked to 
narrative (of hunting) and this form of linkage is now returning to the forefront, due 
to the facilities of digital media. What has been created provides insight into the 
cultural context and social environment at the time of creation. There is an interplay 
between the goal of the creator, the selection and use of appropriate tools, and the 
materials and representations chosen. Gombrich argues that, There really is no such 
thing as Art. There are only Artists [1]. Thus, although creative works may generate 
a variety of interpretations in the minds of observers, the focus also remains on the 
creator, and the vision and insight that causes them to create. 

The symbiotic relationship between art works and the cultural context in which 
they are produced is examined. Technology can provide continuity by making 
traditional methods and techniques more efficient and effective. It can also provide 
discontinuity by opening up new perspectives, paradigms and dimensions of 
interaction with the viewer. This can produce a greater understanding and expan- 
sion of artistic processes and how they are implemented in practice. 
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xiv Preface 


This book includes the arts in general, so is not limited to the visual arts. The arts 
may be defined as the theory and physical expression of creativity found in human 
cultures and societies. A key component of this is communication and narrative. 

This book brings together a variety of national and international authors who 
present current research and development at the interface between technology and 
the arts and humanities. The chapters are grouped into themed parts as follows: 


Part I: A Panoramic View of the Field 

Part II: Facilitating Communication Between the Arts, Technology, and Audiences 
Part III: Interactions Between the Arts and Data 

Part IV: Audio Visual Installations to Generate Collective Human Responses 
Part V: The Convergence of Digital Design, the Arts, Computing, and the 
Environment 

Part VI: The Use of Virtual Reality and Augmented Reality to Extend Creativity, 
Reach, and Engagement in the Arts 

Part VII: The Future of Interdisciplinary Research 


These parts present an increasing reach across the interface using the following 
modalities—communication, interaction, installation, convergence, and the uti- 
lization of VR and AR technologies. The book concludes with an assessment of the 
extent to which the current opportunities and challenges are being addressed and 
realized. 

The convergence of IT, telecommunications, and media is bringing about an 
explosion of data and also a revolution in the way information is collected, stored 
and accessed, plus the major scope of enhancement into time-based media. There 
are three principal reasons why this is happening—reducing cost, increasing quality 
and increasing bandwidth. This is likely to result in a closer relationship between 
technology and the arts and humanities. 

Where the references are to online papers and documents, the authors have 
endeavoured to provide those that are open source and in the public domain rather 
than behind a paywall. The current move to a requirement for open source publi- 
cation in Europe will assist this situation in the future (on the basis that the taxpayer 
has already contributed to the funding of research and development and therefore 
should be entitled to read the publications without further charge). Where publi- 
cations are currently behind a paywall, readers can normally read the abstract and 
see the list of references before deciding whether to purchase the paper. 

There are many references to online sources on the Internet. Readers of the 
e-book can access these directly as they are embedded in the text as hot links. Some 
URLs of web pages change over time due to site names being changed by their 
owners, or the position of the web site in the site hierarchy being altered. Where the 
link does not access the required page, the correct page can often be located by 
putting the URL into Google. If this doesn’t work due to Google’s cached copy 
of the original website having been over-written, then the title of the reference can 
be typed into Google. 


Preface Xv 


This volume is published as a Springer Open Access book in order to make the 
e-book freely available to everyone for study and further research, particularly 
students who may be least able to afford the normal cost of a book. Although the 
e-book contains colour figures, these are rendered in monochrome in the printed 
version, as this is current Springer policy. Therefore, because many of the colour 
figures have been produced by artists, these are best viewed in the e-book. There is 
a nominal charge for the printed book to cover the cost of printing. 

It is hoped that this book makes a useful contribution to an important area of 
significant ongoing research, development and application. 


Leeds, UK The Editors 
Wrexham, UK 

Bradford, UK 

Lausanne, Switzerland 

December 2019 
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Part I 
A Panoramic View of the Field 


Chapter 1 A) 
Introduction and Background geai 
to Technology and the Arts 


Rae Earnshaw 


Abstract Interactions between disciplines are reviewed. The long history of the 
relationship between the arts and sciences is summarized, and the challenges at the 
interface are outlined. The historical developments between technology and the arts 
are summarized, including computer arts, computer animation, digital media, and 
digital humanities. The current enablers for progressing interdisciplinary collabora- 
tions are presented. The possibility for a new Renaissance between technology and 
the arts is discussed. 


Keywords Forms of knowledge - Scientific revolution - Cultural context - 
Computer arts - Story-telling - Reciprocal relationships - Renaissance teams 


1.1 Interactions Between Disciplines 


Interactions between disciplines have always had the potential to be exciting and 
ground-breaking. It can take the participants on all sides into new areas, often uncov- 
ering new understandings and new forms of knowledge. However, cutting across the 
boundaries of disciplines can take researchers out of their traditional comfort zones, 
and can be uncomfortable and challenging for all parties. It has been assumed in 
the past that it is a priority for a particular discipline to protect its legacy, its areas 
of interest, and the particular body of knowledge that it has established and would 
claim as its heritage. This is the foundation upon which new knowledge is expected 
to be built. This been mirrored to some extent by the structures set up in the academy 
to study and disseminate knowledge and research in the disciplines. This foundation 
built over many years can result in inertia and a wish to maintain the status quo. 
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However, new disciplines can arise at the boundaries between existing disciplines. 
Examples are oceanography, cognitive science, genetic engineering, tribology, and 
digital media. Developments and advances such as these have focused attention on 
the interface areas between disciplines, and they are increasingly being recognized 
as important for support and investment. 


1.2 Relationships Between the Arts and the Sciences 


There has been a long history of antipathy between the arts and the sciences due 
to different modes of discourse, different forms of language, and different ways 
of working. This appears to have been caused principally by the early forms of 
education from the foundation of the academy until the scientific revolution from the 
sixteenth to the eighteenth centuries. Prior to this revolution, many understandings 
of the natural world were often misguided and erroneous because they were not 
based on a systematic and rigorous methodology. It is understandable therefore that 
attention in the academy should concentrate on what was regarded as the development 
and disciplining of the mind by means of subjects such as grammar, rhetoric, and 
logic. Later on, physics, metaphysics, and moral philosophy based on an Aristotelian 
framework were also included. Mind took the pre-eminence over matter. The natural 
world was regarded as an environment for the use of tools and the work of tool-smiths, 
and therefore not suitable for academic study. It was not until the key discoveries 
of the scientific revolution that a systematic enlargement of the curriculum in the 
academy took place, to produce the arts and the sciences as we have them today. 


1.3 Historical Developments in the Relationship Between 
Technology and the Arts 


1.3.1 Computer Arts 


The Computer Arts Society [1] in the UK was founded in 1968. Its objective is to 
promote the creative uses of computers in the arts and culture. It acts as a forum to 
bring together those with interests in the cultural impact of information technology, 
and the various ways this impact can occur. This can involve those who create cultural 
artefacts by information technology or manage collections or those who are seeking 
to interpret and understand the cultural implications of the artefacts. An archive of 
the collections of the Computer Arts Society is hosted by the Victoria and Albert 
Museum in London, UK, and is part of their Computer Art Collections. Mason [2] 
and Brown et al. [3] detail the early history of computers and the arts up to 1980. 
Franco [4] detailed the work of Edmonds on generative systems art from the 
1960s to the present day. This explored the relationship between art and computer 
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technologists in terms of concepts, tools, and forms of art. Candy et al. [5] reviewed 
the history of art and technology collaborations highlighting the contributions of 
practitioners and researchers at the interfaces between technology and the arts. 

In the UK in the 1980s, Lansdown and Earnshaw brought together the output from 
the joint work of the Computer Arts Society and the Displays Group of the British 
Computer Society. This work consisted of contributions to visualization, computer 
art, design, and animation [6]. It was recognized that the boundaries between these 
various disciplines were blurring due to the increasing power and capability of the 
computer and the facilities of software packages, with interfaces that were more 
accessible and user-friendly for arts users. 

Computer Art and Technocultures was a 3-year project supported by the Arts and 
Humanities Research Council (AHRC) in the UK to study the history of computer- 
generated art. The project was based jointly at Birkbeck College and the Victoria 
and Albert (V&A) and was completed in August 2010. A display at the V&A on 
Digital Pioneers was exhibited from December 2009 until April 2010. An associated 
symposium, Ideas before their Time [7], was held at the British Computer Society, 
and a two-day conference, Decoding the Digital [8], at the V&A on 4-5 February 
2010. Dodds [9] and Beddard and Dodds [10] provide an account of the V&A’s 
collections and their relationship to art history, and their social and technological 
context and implications. It also examines the outputs of the AHRC project. These 
are also summarized as part of the project [11]. 


1.3.2 Computer Animation 


Making images move has been a preoccupation of computer technologists from 
the earliest days of computers [12]. Prior to this, successive frames of an ani- 
mated sequence had to be composed on print media and then filmed, which was a 
time-consuming process. With the advent of computer displays and software, image 
sequences could be created and displayed directly. This facilitated ease of editing 
and re-display, which was more time consuming with print-based media. 

Today there is a wide variety of desktop software packages that enable home 
users to create animations with audio tracks. Vince [13] discusses animation tech- 
niques, animation hardware, and animation software such as Softimage, Maya, 3d 
Studio Max, and Lightwave. Post-production techniques are presented, and anima- 
tion applications are reviewed. Such Computer-Generated Imagery (CGI) has found 
widespread use in the applications such as special effects in films, computer games, 
advertising, computer-aided design, and simulation. However, computer technology 
of itself does not provide an automatic or easy solution. The imagination and creativ- 
ity of the person formulating the animation need to be the fundamental driving force. 
The hardware and software technology are just tools and need to be used appro- 
priately [14]. Animated films such as Pixar’s Toy Story [15], which won awards, 
illustrate this point. It is the story which draws in, and captivates, the viewer. There- 
fore, in order to be effective artistically, animation sequences are designed and chosen 
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to best represent the story. Story-telling is itself an art and has a long history [16], 
whether in oral form or using various forms of media that were available at the time 
in earlier centuries. 


1.4 Digital Media 


Collaboration in the production of digital media may involve a variety of discipline 
specialists. This is because of the diversity of its constituent parts such as hard- 
ware, software, digital images, digital, audio, sensors, games, interaction devices, 
and social media. This in turn may involve different constituencies such as indus- 
try, the academy, research and development organizations, and Small and Medium 
Enterprises (SMEs) with all their different cultures and working practices [17, 18]. 
The challenges are therefore significant. However, for those with a good collabo- 
ration methodology geared to generating successful outcomes, the rewards can be 
substantial. 


1.5 Digital Humanities 


On the wider front, the humanities disciplines such as history, literature, and phi- 
losophy are increasingly using computational tools and facilities to advance their 
research. Digitized texts can be searched, word frequencies can be calculated, and 
indexes and concordances can be produced [19]. In addition, image processing and 
3D recording can be useful for pictorial data and 3D spaces such as museums and 
heritage sites [20]. Two broad areas of digital humanities research may be identified: 
firstly, the use of digital tools to perform and extend research in the humanities and, 
secondly, the use of humanities applications to perform research in computer sci- 
ence in the development of new interfaces and more advanced tools. Thus, there is 
a mutual reciprocity between technology and the humanities, even if this is not ini- 
tially recognized [21]. The cultural context of computing also needs to be taken into 
account when evaluating its contributions. Such interdependencies are of increasing 
importance when seeking to understand current interdisciplinary activity and define 
potential ways forward for the future. 


1.6 A New Renaissance? 


The Renaissance in the fourteenth century onward marked a transition from the 
medieval era to modernity, and it opened up the possibilities of new horizons and new 
ways of thinking and working. Similarly, Renaissance Teams [22] in the twenty-first 
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century offer the opportunity to utilize the expertise now available and work together 
toward a common objective which can transcend the traditional boundaries of the 
past. 

This point is illustrated and exemplified by the contributions to this book. 

This book brings together a variety of national and international authors who 
present current research and development at the interface between technology and 
the arts and humanities. The chapters are grouped into themed sections as follows: 


Section 1: A Panoramic View of the Field. 

Section 2: Facilitating Communication between the Arts, Technology, and 
Audiences. 

Section 3: Interactions Between the Arts and Data. 

Section 4: Audio Visual Installations to generate Collective Human Responses. 
Section 5: The Convergence of Digital Design, the Arts, Computing, and the 
Environment. 

Section 6: The Use of VR and AR to extend Creativity, Reach, and Engagement 
in the Arts. 


These sections present an increasing reach across the interface using the following 
modalities—communication, interaction, installation, convergence, and the utiliza- 
tion of VR and AR technologies. The book concludes with an assessment of the 
extent to which the current opportunities and challenges are being addressed and 
realized. 

The convergence of IT, telecommunications, and media is bringing about an explo- 
sion of data and also a revolution in the way information is collected, stored, and 
accessed. There are three principal reasons why this is happening—reducing cost, 
increasing quality, and increasing bandwidth. This is likely to result in a closer 
relationship between technology and the arts and humanities. 
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Chapter 2 A) 
Positioning the Arts in the Research ciecie; 
Process: Perspectives from Higher 

Education 


Susan Liggett 


Abstract Research in the visual arts has contributed to the creation of environments 
that involve cross-disciplinary, multidisciplinary and interdisciplinary or transdisci- 
plinary projects in departments within and across universities. An overview is pro- 
vided of the historical context of doctoral awards in the arts with a definition of the 
terms practice-based, practice-led, and practice as research discussed. It articulates 
the challenges when acquiring explicit and exact knowledge alongside more subjec- 
tive approaches that utilize tacit knowledge from artistic practice in research projects. 
Drawing on examples from art practice and doctoral students work, it analyzes objec- 
tive, subjective, empirical, and hermeneutic paradigms, as described by Pierre Bour- 
dieu, which can combine empirical approaches and individual understandings to 
re-enforce our understandings of the world. 


Keywords Practice-based research - Practice-led research « Practice as research 
(PaR) creative arts Ph.D. + Collaboration - New technologies + Cross-disciplinary + 
Interdisciplinary 


2.1 Introduction 


Most researchers are aware of the need to position themselves in relation to other 
fields or disciplines in order to reveal the particular characteristics of their research 
findings. At the start of the research process, usually in writing the proposal, it can be 
challenging for artists when “oneself” often plays a significant role in the object of an 
enquiry. Research in the arts can be largely hermeneutic; that is the understanding and 
interpreting of it exposes the subjective limits of the artists’ ways of doing and seeing. 
It is this epistemological method of socially constructing meaning that gives the arts 
authority within a process of enquiry. However, it is all too easy for knowledge 
gained tacitly to be “overlooked because it is subsumed into the rational logic of 
discursive accounts of artistic production” [1]. The articulation of possibilities of 
artists for shaping consciousness and providing cultural capital can be difficult even 
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for the more experienced academic. How can, as Barrett [1] says, “the interplay 
of disparate areas of knowledge create new analogies, metaphors and models for 
understanding objects of enquiry”? 

Cross-disciplinary research involves viewing one discipline from the perspective 
of another. Interdisciplinary research involves integrating knowledge and methods 
from different disciplines that use a synthesis of approaches and multidisciplinary 
research that involves people from different disciplines working together, each draw- 
ing on their disciplinary knowledge. Transdisciplinary is close to multidisciplinary 
but it moves beyond a discipline taking more of a holistic approach creating a union 
of intellectual knowledge that goes beyond the disciplinary perspectives. 

From experience of supervising doctoral students in the creative arts, I have 
noticed that an artist’s identity can be lost when research methods are combined 
from different disciplines, such as those from the social sciences where is it easier to 
articulate the possible new knowledge that may emerge from interviews or surveys, 
for example, than those resulting from the art practice. This is particularly evident 
when the student begins to scrutinize artwork in unfamiliar ways to seek valida- 
tion from research communities in different disciplines. For example, at Wrexham 
Glyndŵr University all research students are required to present their work to the 
whole research community in an event called “Open House for Researchers”. Art 
students need support when combining different methodological paradigms in their 
research to bring fresh perspectives and new possibilities to problems. Artists in 
academia need to convince practitioners from other fields of enquiry within and 
beyond the university that they can reflect new objective realities that may be of 
benefit to society. 


2.2 Artistic Research and the Academy 


It is useful to decode some of the terminology surrounding the arts in academia to 
facilitate analysis of what artists do in a meaningful way. The word “practice” is used 
in the arts as it is in other professions to recognize knowledge production through 
action and doing. We can only fully understand what happens when we make artwork 
through the process of doing it, as opposed to purely thinking about it and the way 
artists create work goes beyond the physical activity to include influences, ideas, and 
critical reflections. The term practice refers to all artists work, but not all art practice 
can be termed research in an academic context. Despite the fact that most practice 
involves some form of research as part of the process, for example taking photographs 
in preparation for a painting, is not considered research in an academic context if 
the resulting work is not shared and its contributions to knowledge challenged. The 
theory/practice nexus helps define research in the arts by the linking of theory and 
practice through a method of intentionally investigating the process and outcomes 
informed by practice. The term “praxis” is often used to describe the making of 
artwork in a research context. Practice without the theory remains “practice” which 
can get very confusing for those involved in the arts, and also for those from the 
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outside looking in. However, the adoption of the term practice is clear and implies 
that the production of artworks is part of society and connected to lived experience 
rather than existing in an ethereal world empty of everyday concerns and therefore 
unimportant and insignificant. Integrating the arts in society forces us to articulate 
how the purpose and meaning of art is constructed, challenged and its investments 
accounted for. 

Bringing the arts into academia gives it status and legitimizes it as a profession. In 
the UK the first training in the arts was available in 1768 when The Royal Academy 
of Arts opened; art schools were further developed in the nineteenth century and 
re-designed in post-war Britain under the influence of the Bauhaus [2]. In 1974 
in the UK the National Council for Diplomas in Art and Design was merged into 
the Council for National Academic Awards (CNA) allowing Polytechnics with art 
departments to issue honors degrees for the first time. Further academization of the 
arts happened in 1992 with former polytechnics becoming Universities [3]. This put 
the arts on an equal footing with other subject areas historically considered more 
academic. With the home of the Art School now residing within Universities, artists 
have the opportunity to study for research degrees and there has been an exponential 
growth in uptake of doctorates in the arts over the last decade. Between 1986 and 
1995, 181 students received research degrees in the arts subjects in the UK [4] and 
in 2016/17 alone, 880 students were awarded research degrees in art and design 
(HESA) [5]. 

The Ph.D. is now a standard requirement for teaching in an art department in a 
UK University and with this, it brings pressure on academics to compete for research 
funding in ways comparable to other academic subjects. With the arts being newcom- 
ers to universities system, ontological truth claims between the Realist! worldview 
and the Constructivist? world view are challenged with artists claiming that often 
regulatory frameworks within Universities consist of a strong Realist components 
with the use of the terms “question” and “answer” rather than the terms “issue” and 
“response”. 

If the arts do indeed operate within a new paradigm they should be able to say: our ontological 

position is this, our epistemological position is this, our methodological position is this and 


all of these are coherent and this is why we warrant special different conditions to the ones 
that have formally been recognized [6]. 


Art departments have had to fight hard to ensure that systems were set up to support 
their students in practice-based research. Between 2000 and 2006 debate surrounding 
the doctorate in the arts was highlighted at the University of Hertfordshire’s annual 
conferences on the foundations of practice-based research which published Working 
Papers on Art and Design [7]. Particular concern regarding academic standards being 
imposed on art practices and artistic research were further addressed internationally 
[8] with European League for the Institute of the Arts (ELIA), with the creation of an 


l Realist view here refers to the idea that reality exists independently of observers and their thoughts, 
feelings, intuitions, and opinions. 

Constructivism her is recognition that reality human construct that interacts with our experience 
in the real world. 
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advocacy network to address issues facing the arts in higher education. Its aim is to 
collaborate to create networks that strengthen, support, promote, and encourage arts 
institutions across Europe. A specific working group to enhance doctoral research in 
the arts known as the SHARE? network holds annual events to raise the platform for 
research in the arts. 


2.3 Knowledge Production 
and Practice-Based/Practice-Led Research 


Practice-based, practice-led, practice as research (PaR), artistic research, and arts- 
based research are all widespread terms in creative arts research which are not inter- 
changeable and exhibit qualities that need to be understood regarding the nature of 
the art practice and the resulting production of knowledge. Different perspectives 
on these can be explored through the writings of Candy and Desmond [9, 10], Bar- 
ratt and Bolt [11], Grey and Malins [12], Macleod and Holdridge [13], Briggs and 
Karlsson [14], Nelson [15]. 

In summary, “Practice-based Research is an original investigation undertaken 
in order to gain new knowledge partly by means of practice and the outcomes of 
that practice”. By contrast “Practice-led Research is concerned with the nature 
of practice and leads to new knowledge that has operational significance for that 
practice” [16]. Practice as research (PaR) distinguishes artist-scholars in a university 
context from professional practice. Nelson [15] describes PaR as a research project in 
which practice is a key method of enquiry where the practice is submitted as evidence 
of a research enquiry. An important difference between the personal or professional 
artist and artwork in the doctoral research is the form the knowledge produced takes. 
For example, “understandings about audience experience, taxonomies, models of 
collaboration and the artwork themselves” [17]. This is not to be confused with 
arts-based research which is another term used principally in the fields of education 
where it is used to understand education through arts-based concepts, techniques, 
and practice [18]. A practice-based Ph.D. includes creative works that arise from the 
research process as part of the submission. 

Despite the literature available on different perspectives on practice-based 
research, there is still not an “an integrated discourse on the place of practice-based 
research in Ph.D. programmes” [17]. The definition of research within the univer- 
sity is clear and Wrexham Glyndwr University had articulated it as a “process of 
investigation leading to new insights collectively shared” [19]. Two expectations of 
new knowledge resulting from by research are firstly, that it has to be open to being 


3SHARE stands for Step-change for Higher Arts Research and Education and was an international 
networking project, comprising 39 partners working together on enhancing the “3rd cycle” of arts 
research and education. It created a Europe-wide exchange framework for the widely different 
experiences, practices and ideas that make up the lively domain of artistic and cultural research. 
[https://www.elia-artschools.org/activities/artistic-research/share]. 
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challenged and, secondly, that it has to be verified. Creative arts research outcomes 
and methodologies are sometimes difficult to understand and quantify in terms of 
traditional scholarship, which can lead to a devaluing of studio-based enquiry [20]. 

It is only when artist researchers, with their practice, firmly root themselves in 
their discipline, and position themselves in relation to other disciplines, that they 
can then truly reveal the characteristics particular to their research. Students enrolled 
on Ph.D. programmes are frequently expected to present their research to the wider 
community of students from other disciplines. This can have a “de-centering” effect 
on the newly enrolled student. On writing the research proposal it can, initially, be 
confusing if “oneself” plays a significant role in the object of an enquiry. This has 
also presented challenges to supervisors at Wrexham Glyndwr University guiding 
students in cross-disciplinary teams. 


2.4 Cross-Disciplinary, Interdisciplinary, 
and Multidisciplinary Collaborative Research 
in Doctoral Studies 


Bourdieu notes that on-going privileges of positivistic and instrumentalist approaches 
to research persist [20, p. 4]. The researchers in the case studies described have 
experienced a de-stabilizing effect on their confidence when writing proposals. A 
lack of experience sometimes leads to an over reliance on scientific or social science 
paradigms to justify “fuzzy” concepts. 

The following examples demonstrate a research project conducted by Liggett and 
two that she has supervized to illustrate how these challenges have been overcome 
by researchers having a clear understanding of themselves and their motivations for 
the research. They all adopt a mixed-method approach and involve interdisciplinary 
or cross-disciplinary collaborations through the research design of the supervisory 
teams comprising academics from different subject areas. Supervisors recognized 
the importance of openness and a willingness to seek the advice of other academics 
to ensure the support was provided for cross-disciplinary dialogues in the following 
projects. 

Firstly, Heald and Liggett undertook a multidisciplinary research project in col- 
laboration with a Consultant psychiatrist and Professor of social psychiatry both 
from Bangor University titled In-between-ness: using art to capture a sense of self 
during anti-depressant treatment [21]. The rationale for the research was to try and 
understand how reality is constructed and whether this process can be manipulated 
with medication and also, to determine the implications for the authenticity of “self” 
for someone undergoing treatment for depression. The artists worked one-to-one 
with four participants over 6 weeks. Service users were issued with a video camera 
and asked to film on a weekly basis with visual prompts introduced as a stimu- 
lus. The process allowed participants to explore how views of themselves changed 
as they recovered from depression. Interviews and psychometric tests were carried 
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out before, during and after their treatment, in collaboration with the trial psychia- 
trist. The findings revealed that the interview transcripts recorded positive benefits 
to participants in terms of increased responsiveness to sensations, their surrounding 
environment, the quality of their feelings and their sense of “self” when engaged in 
this creative process. The psychometric tests, by contrast, evidenced no change in 
data. 

The artists’ practice acted as a personal rationale in the approach to the research 
and initial research questions stemmed from the issues arising from their work. 

These questions were recognized by the medical professionals and validated by the 
psychiatrists in the collaboration. This external rationale then drove the art practice 
as the artists made new artworks in response to the situations and experiences they 
encountered on their research journey. Figure 2.1 shows Liggett and Heald’s film 
work made in response to feeling “in-between” working as artists at a psychiatric 
unit. The artwork cannot be detached from the research project, and the artwork 
produced by participants was seen as integral to the project and exhibited alongside 
the artists’ work in the final exhibition concluding the project. 

Secondly, Braisby’s Ph.D. project, which was practice-led and interdisciplinary 
[22], identified how galvanic processes can enhance the work of the artist printmak- 
ers and act to reverse the current decline in the teaching of intaglio etching. His 
work involved making artworks to gain a deeper understanding of the chemical and 
electrochemical processes involved. Scientific advice was sought from academics 
in the chemistry department and workshops were designed for artists to learn about 


Fig. 2.1 Paper Interior, film duration 9’ 20”, Heald and Liggett (2013). Photo credit Karen Heald. 
Copyright © K. Heald, 2019 
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the processes. The learning generated from the workshops was analyzed using Soft 
Systems Methodology (SSM) developed by Checkland [23]. It involved drawing a 
rich picture; a process that uses images and words drawn out on a large sheet of paper 
to describe the problem situation. Exposing the richness of the data helped in the 
framing and re-framing of the situation and generated new ideas and insights into 
the problem situation. 

This rich picture process identified two aspects to the research question. The 
first relates to the research into the galvanic techniques and the second to their 
application as an artistic medium. The first is a well-defined problem that can be 
addressed through a scientific problem-solving experimental approach under the 
rubric of the “hard systems” [23]. The second part of the question needed to engage 
with messy, real-world issues. This world was engaged through: workshops (electro- 
etching), presentations (using the rich picture), collaborative work (other artists), and 
interviews. 

The research strategy developed was based on a hybrid model of both the practice- 
based and practice-led methods [16]. The quantitative (practice-led) research col- 
lected data by experimentation and developed a new process to produce the artwork. 
The qualitative data (practice based) developed new understandings of the process. 
Figure 2.2 shows experiments in enameling or plating etching plates with a differ- 
ent metal in its impact on the practice of etching was evident in the creation of the 


Copy of part of plate enamelled Copy of part of plate enamelled Copy of part of plate enamelled 


Fig. 2.2 Don Braisby, Experimenting with Enamelling (2019) 23 x 29 cm Copyright © D. Braisby 
2019, reproduced by permission 
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artwork produced. The research required the researcher to move out of his artistic 
environment and engage with chemists, scientists, and physicists. 

Finally, Wyatt’s Ph.D. cross-disciplinary project [24] investigated how people 
living with dementia engage with, and experience, painting whilst working alongside 
an artist-researcher. Arts-based inquiry* was adopted as a method creating a body 
of paintings to provide insights into different experiences and forms of engagement 
when painting. Figure 2.3 is an example of a painting Wyatt made in response to 
the memory of a beach in Wales. These were used to inform a qualitative study that 
used social science methods of interviews observations and video recordings. The 
use of both arts and social science research methods presented a challenge initially 
but the research demonstrated how an artists’ perspectives can contribute to the field 
of health by promoting new approaches to research. 

In all three projects, the subjective strength of their work has grown out of the 
positioning of the researchers in their studies by a self-scrutinizing process that is, 
perhaps, unfamiliar in other disciplines. This process is necessary to prevent the artist 
becoming a “pseudo-scientist’, having a “split personality” or being over narcissistic. 

The in-between-ness project involved a collaborative multi-method pluralistic 
approach to the research questions and monthly dialogues within the research team. 
Initially, the challenge for the artists was to emphasize their individual practice in 
the research design. Ostensibly their role could be seen to be purely one of facilitator 


Fig. 2.3 Megan Wyatt, Whistling Sands (my Favourite Beach), 2017 Copyright © M. Wyatt 2019, 
reproduced by permission 


4 Arts-based inquiry uses artistic expression as data for inquiry. 
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for the participants. In reality, it was much more than this. The collaborative process 
offered the artists a new method of challenging ideas that grew out of their practice. 
It wasn’t until each researcher’s motivations and position within the research process 
was established that a meaningful collaboration existed where all academics equally 
owned the research. 

In the “In-between-ness” project with mental health participants and Wyatt’s 
project involving people living with dementia the relationship between the researcher 
and the participants is an intrinsic aspect of the research. When viewing and making 
artwork, it is not merely the deciphering of reality which occurs, because new under- 
standings emerge, and connections and explanations are exposed to both viewer and 
artist. Viewing and creating artwork can transform understanding to inform and gen- 
erate new and valid knowledge [25]. Practice-based research is used in these projects 
as it allows for new outcomes which do not oppose a verbal explanation [26]. 

In Braisby’s research involving galvanic etching, the known data from multiple 
points of view is gathered together and made visible. This data enabled the researcher 
to identify hidden links and make new connections and insights with their personal 
practice to facilitate the refinement of the research. This process exposed the richness 
of the data that emerged from the research and helped in re-framing the situation. 
This firmly positioned the researcher in this research field, and this practice generated 
new ideas and insights into the problem of why electro-etching has never become 
mainstream in institutes of art. 


2.5 Art, Collaboration, and Technology 


The use of art as tool for social change is gaining traction with advocates such as 
Alistair Hudson, the director of the Whitworth and Manchester Art Gallery since 
2017, which promotes projects that have a real impact on people’s lives. People need 
to value art so that they no longer say that they need a hospital rather than a gallery. 
The mission is to get people to understand that the two are not mutually exclusive 
and that people need both hospitals and art galleries [27]. 

According to the World Economic Forum, creativity is a key skill needed for 
future employment [28]. The Fourth Industrial Revolution [29] brings advances in 
technology that are challenging traditional jobs in the shift from the human to the 
digital. Robots and artificial intelligence (AI) are more efficient in certain areas than 
humans in the workplace. In the future, 47% of work may be lost to automation pro- 
cesses [30]. Mobile Internet and cloud technologies are impacting society. However, 
robots do not yet possess innate creativity (though software may soon be able to 
emulate some of its characteristics) so business leaders, educators, and governments 
still recognize the importance of human creativity. 

Artists are increasingly collaborating with scientists and technologists in a multi- 
tude of ways that extend the boundaries of knowledge. For example, The Foundation 
for Art and Creative Technology (FACT) a leading UK visual arts organization has 
developed a number of projects promoting creative media and creative technology. 
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In 2019, it worked with the NHS, BBC, charities, and businesses in a project called 
“The Future World of Work” to provide an arts-led critique of innovation and tech- 
nology, reimagining work in relation to gender, the gig economy and careers advice 
for uncertain times [30]. It worked with a number of artists who undertook workplace 
artist residencies who make artwork in response to their experiences culminating in 
an exhibition at FACT’s gallery [31]. 

Artists can provide useful critiques of society and can raise awareness of societal 
issues in various ways. The German artist and filmmaker Hito Steyerl has an interest 
in how the media, technology, and the global circulation of images impact society. 


We live in a world that is already deeply edited and PhotoShopped. It is cut and pasted, and 
the people who know these practices because they work with them everyday — let’s call them 
artists — they understand the importance of these practices, not only understanding the world 
but also making it as it is [32]. 


Steyerl’s video installation Factory of the Sun examines the desires and threats 
of image circulation and the possibilities of shared resistance when surveillance is 
everywhere in our increasingly virtual world. Factory of the Sun tells the surreal story 
of workers whose forced moves in a motion capture studio are turned into artificial 
sunshine [33, 34]. 


2.6 Conclusions 


The artists’ subjective and hermeneutic approaches offer new perspectives to research 
but there are challenges in articulating the unique qualities their work offers. Cross- 
disciplinary supervisory teams at Wrexham Glyndwr University have enabled critical 
dialogues for the students and supervisors to set out clearly and test the concise 
methodological approaches to their work. The unpredictability of artistic research 
has brought challenges, particularly when adopting a less prescriptive approach to 
outputs in the research proposal. The researchers here often found it easier to talk 
about the results of their interviews, or scientific experiments, than articulating the 
impact of their practice within their projects when applying a multi-method approach 
to research questions. In the In-between-ness project, for example, the therapeutic 
effect of the project was not the primary measuring stick for assessing the success 
of their work. 

According to Polanyi (1962), aesthetic insights “motivates the early stages of 
much scientific research” [35]. Mixed disciplinary supervisory teams at Wrexham 
Glyndwr University have forced us to challenge perceptions of ourselves as artists 
and consequently enriched our projects. We often cannot predict what others will 
learn from our work, but it is the process of learning and identifying one’s individual 
position in the research process that will allow for new avenues of knowledge to 
emerge. The arts can be a powerful tool for social change, for wellbeing and for 
connecting people. As Candy states in the summary of her new book, there is a 
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considerable amount we can learn from paying close attention to the world and it 
takes a creative mind to do this with insight [36]. 
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Part II 
Facilitating Communication Between 
the Arts, Technology, and Audiences 


Chapter 3 A) 
Framing the Conversation: The Role rie 
of the Exhibition in Overcoming 
Interdisciplinary Communication 

Challenges 


Susan Liggett and Mike Corcoran 


Abstract The role of the exhibition in overcoming interdisciplinary communication 
challenges is explored. The meaning of interdisciplinary communication is estab- 
lished, alongside the challenges it brings, where their route causes lie and the extent 
to which the exhibition is equipped to address them. It concludes that exhibitions of 
a particular nature are uniquely equipped to facilitate better quality interdisciplinary 
communication in certain contexts, but that further work is needed to develop mature 
curatorial models to facilitate this. A structure for such models is proposed, along- 
side the steps that now need to be followed if exhibitions are to fulfil this untapped 
potential at the interface of art and technology. 


Keywords Curatorial model - Communication - Exhibition - Interdisciplinary - 
Art + Technology 


3.1 Introduction 


Interdisciplinary solutions are becoming the lingua franca in the discussion of the 
global challenges we face in the twenty-first century. Whether it’s the education of 
our young people, the care of our elderly, the fight against disease or the protection 
of our environment, the answers will not be found in any one place. Rather, progress 
relies on the coming together of diverse expertise and perspectives, and the shared 
commitment of a vast number of individuals, unified by a common goal. 

At the meeting point of art and technology, this is felt more sharply than perhaps 
anywhere else. The ‘fourth industrial revolution’ has blurred the boundaries between 
man and machine, with the designer’s role fundamental in managing the interface 
between the two. Furthermore, artists have ever-increasing responsibility for bring- 
ing the challenges of our times to the masses in ways which are tactile, accessible 
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and engaging to all: driving political and social agendas. Conversely, technology is 
opening up endless new ways to create, display and interact with art, spawning new 
branches of artistic practice, and changing our perceptions of what art is and what it 
can achieve. 

Communication is central to the success of any group, working towards any objec- 
tive. When groups are interdisciplinary, subject matter is technical, stakeholders are 
global and success is critical, this presents unique challenges. However, in spite of 
this, the nuance of communication in such contexts is still poorly understood. Exhi- 
bitions are a unique communication tool. Their potential as a means to support better 
quality communication amongst interdisciplinary teams is apparent; yet this potential 
is under-investigated. The processes by which it could best be realized are ill-defined, 
and the models and tools required to facilitate the execution of such processes are 
absent. 

Here, a forensic examination of communication amongst interdisciplinary art- 
technology teams is undertaken to establish the root causes behind the challenges 
which arise and the extent to which exhibitions are equipped to provide solutions to 
them. Assumptions are tested against the experiences of the principal actors in such 
communications and the way forward is posited, with proposals set out for the further 
work required if exhibitions are to fulfil their potential in improving interdisciplinary 
communication: for more people, more of the time. 


3.2 Defining Interdisciplinary Communication 


3.2.1 Defining Interdisciplinary 


Before we identify interdisciplinary communication challenges, we must first 
establish what is meant by interdisciplinary and communication. 

The terms interdisciplinary, multidisciplinary and transdisciplinary and applied 
inconsistently, and used interchangeably by many [1]. However, the National 
Academies provide the following working definition of interdisciplinary research: 
“a mode of research by teams or individuals that integrates information, data, tech- 
niques, tools, perspectives, concepts, and/or theories from two or more disciplines 
or bodies of specialized knowledge to advance fundamental understanding or to 
solve problems whose solutions are beyond the scope of a single discipline or area 
of research practice” [2]. 

It is in this sense that interdisciplinary will be referred to herein: referring to the 
coming together of teams or individuals from two or more bodies of specialized 
knowledge in research, or otherwise. 
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3.2.2 Defining Communication 


In order to precisely identify communication challenges, it is helpful to adopt a sim- 
ple conceptual model of human communication which clearly delineates each of 
its component parts. The Shannon—Weaver Model [3] was developed by Bell Lab- 
oratories engineers Elwood Shannon and Warren Weaver in the 1940’s. It is widely 
regarded as the forerunner of modern communication studies and provides such com- 
ponents. Shannon and Weaver extrapolated a general theory of communication from 
the process by which radio technologies function and identified five components of 
all communications, as represented in Fig. 3.1. 

An Information Source produces the message which is to be communicated. 
A Transmitter encodes that message into appropriate signals for transmission. A 
Channel provides the medium through which signals are transmitted (and as signals 
propagate through this Channel, they are vulnerable to ‘Noise’ which can block, 
interrupt or distort signals). A Receiver decodes the signal to reconstruct the original 
message, and a Destination is an individual, group or entity for whom the message 
is intended. 

Consider saying hello to a friend as you walk down the street. You, or perhaps 
we should say, the appropriate regions of your brain, are the Information Source, 
taking the decision to transmit the message, ‘Hello! . Your lungs, vocal folds, tongue 
and mouth are the Transmitter, working together to produce the appropriate sound 
waves and air is the Channel which carries these sound waves between you and 
your friend. Passing buses, sudden gusts of wind and much else can provide noise to 
distort the message at this stage, before your friend’s ear receives the sound waves 
transmitted, and their brain converts and translates them to reveal their intended 
meaning. The message has now reached its intended destination and your friend 


Noise 


Information 


S Transmitter Channel Receiver Destination 
ource 


Fig. 3.1 Shannon and weaver’s five component parts from which all communications are 
comprised. Copyright © M. Corcoran 2019 
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responds appropriately, perhaps with a wave in your direction. Through this new 
transmission (and feedback), the iterative process begins again. 


3.2.3 Defining the Principal Actors in Interdisciplinary 
Communication 


Alongside a conceptual model of communication, it is also crucial that we have 
an understanding of who the principal actors (in Shannon and Weaver’s language, 
the Informational Sources and Destinations) in interdisciplinary communications 
typically are. We propose the following groups, as shown in Table 3.1. 

With working definitions of interdisciplinary and communication established, 
alongside an understanding of who such interdisciplinary communication occurs 
between, we can now consider the challenges of interdisciplinary communication 
and their direct relationship to activities involving artists and technologists. 


3.3 Interdisciplinary Communication Challenges 


Shannon and Weaver proposed that, considering the five parts from which all com- 
munication is comprised, three fundamental types of communication challenge can 
arise: 


Table 3.1 The principal actors interdisciplinary communication 
Principal actor | Definition 


Specialists The individuals and teams who carry out the work. They possess knowledge of 
one or more technical discipline and are unified by a common set of objectives 


Benefactors Those who sponsor the work. This includes philanthropic funders, public 
bodies and private investors 


Beneficiaries Those who reap the rewards of the work, including governments, private 
companies and proportions of the general public 


Publics A meta-group that include all those within a given population who are 
stakeholders in an interdisciplinary activity, either as specialists (contributing 
their insights with respect to a particular problem), Benefactors (through the 
public funding of interdisciplinary work through taxation or others) and 
Beneficiaries (in virtue of an activity’s impacts). Often Publics will hold 
multiple roles simultaneously 


Translators Facilitating communication between two or more of the aforementioned 
groups in any combination. For example, they may assist specialists in 
engaging with Publics, Benefactors in evaluating Beneficiaries, or Publics in 
lobbying Benefactors 
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1. Technical Challenges: relating to how accurately messages can be issued. 

2. Semantic Challenges: relating to how precisely meanings can be conveyed. 

3. Effectiveness Challenges: relating to how successful received messages are in 
bringing about their desired effects. 


3.3.1 Communicating Accurately 


Technical challenges arise due to the many different means artists and technologists 
have to adopt to get their messages across. 

The typical interdisciplinary communication involves multiple information 
sources, and multiple Destinations. Diverse information is shared in all directions: 
both within specialist interdisciplinary teams, and between those teams and their 
Benefactors, Beneficiaries, Publics and Translators. Every group (and every actor 
within every group) has their own preference with respect to transmitting and receiv- 
ing information (some like to write or talk, others like to draw or experience and 
so on), and information tends to go through many iterations of transmission and 
reception on its journey between actors, highly subject to noise and interference. 

This is especially true when artists and technologists work together. Consider 
what it takes for the latest research in machine learning to reach the desk of the fine 
art professor or for the biomedical scientist to encounter theories in the teaching 
of painting. Consequently, before the insights of one actor enter the consciousness 
of another, they may be transmitted through many cycles of performance, discus- 
sion, publication or otherwise, with each increasing the probability of degradation, 
mistranslation and dilution of the overall message [4]. Even before interdisciplinary 
teams have been constructed, the basis for their very construction can run the risk of 
being underpinned by inaccurate information. 


3.3.2 Communicating Precisely 


Semantic challenges arise due to the difficulty of explaining things in a simple way 
which artists, technologists and everyone they engage with can understand. 

There is no universally understood language amongst the principle actors in inter- 
disciplinary activities. As Derrick et al. observe, “Scientific disciplines have different 
cultures, languages, and standards” [5] and the same can be said of non-scientific dis- 
ciplines. The vocabulary and reference points of specialists are highly technical, and 
a deep understanding of issues is often contingent upon a contextual understanding 
of the wider subject, not always available to every specialist within an interdisci- 
plinary team. This is particularly true when artists and technologists come together. 
Their respective disciplines having long been siloed into what CP Snow famously 
characterized as ‘The Two Cultures’ [6]. 
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This problem is only compounded as specialists (of any discipline) look to com- 
municate to their Benefactors, Beneficiaries and Publics. Translators can assist, but 
ultimately, communication is often heavily reliant on metaphor and simplification, 
inevitably impacting on the precision with which information can be shared, and 
preventing much information from being shared at all. 


3.3.3 Communicating Impactfully 


Effectiveness challenges arise due to the many different things artists and technolo- 
gists have to simultaneously achieve when they communicate. 

Often, a single communication will have multiple Destinations and multiple 
diverse objectives. As communication takes place amongst specialists within an 
interdisciplinary team, the outcomes and impacts of such work are simultaneously 
subject to the scrutiny of Benefactors, Beneficiaries and Publics, each with diverse 
needs and expectations. To achieve such objectives can be ‘complex, nuanced and 
highly resource intensive’ [4]. When it is simply not possible to tailor the perfect 
transmission for each target Destination, a ‘one size fits all’ approach can be con- 
sidered as an alternative, but in striving to please all, there is an inherent risk of not 
pleasing any. 

Taking into consideration the main components of communication, the fundamen- 
tal types of communication challenge posited by Shannon and Weaver, and reflecting 
on each in the context of the principal actors in interdisciplinary communication, 
especially artists and technologists, we can conclude that such communication is a 
challenge because: 


1. Technical challenges lead to the accuracy of information shared being impeded 
by the many different ways and means required to communicate. 

2. Semantic challenges lead to the precision of information shared being impeded 
by the lack of common vocabulary and reference points amongst the principle 
actors who are communicating. 

3. Effectiveness challenges led to the impact of communication being impeded by 
the multiple objectives it is required to serve. 


In transitioning from an understanding of the challenges which lie at the heart 
of interdisciplinary communication to an understanding of the ways in which these 
challenges can be overcome and the role of the exhibition in facilitating this, it is 
essential to gain a deeper appreciation of the validity of our conclusions above. This 
includes the extent of their generalisability, and the contexts in which they most 
strongly apply. This starts through dialogue with those with lived experience of 
interdisciplinary collaboration from a variety of different perspectives. 
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3.4 Interviews with Interdisciplinary Practitioners 


The following perspectives are derived from interviews conducted with various 
experts and specialists in the domain of this research. That is, those who are primarily 
responsible for the management and administration of interdisciplinary projects and 
activities, and the communication that comes with it. Their thoughts and experiences 
are summarized below. 


3.4.1 The Specialist Technologist’s Perspective 


The specialist technologist’s perspective was offered by a software engineer working 
predominantly in the rendering of digital holograms. 
This involves communicating primarily with: 


e Artists (Benefactor/Beneficiaries): including both independent artists and com- 
mercial customers looking for holographic solutions within their work and 

e Optical engineers (Specialists): in order to understand and provide feedback on 
the strengths and restrictions of working in the medium. 


Communication with fellow technical specialists (i.e., software and optical engi- 
neers) tends to be the most straightforward. This is in part thanks to the common 
vocabulary and reference points these specialists share, and the readily quantifiable 
nature of the information they typically share (for example, the technical parameters 
of the hologram printing equipment). For communication of this nature, communi- 
cation tools such as email are very effective, allowing for precise information to be 
shared back and forth asynchronously. 

Conversely, communication between artists and engineers is more challenging, 
because there is no common vocabulary and the information being shared is typically 
subjective and qualitative. Engineers and artists are often exploring the ‘art of the 
possible’, and the broad objective which the creative work is attempting to realize. 
In these circumstances, live and iterative communication, supported by visual aids, 
can prove effective. 

With regards to exhibitions, they can play a valuable role in setting the context for 
discussions between any actors, alongside demonstrating the nature and potential of 
the technology, especially when bringing such state-of-the-art technology to general 
public audiences. 


3.4.2 The Specialist Artist’s Perspective 


Jessica Lloyd Jones, an artist who exhibits internationally combining elements of art, 
science and technology provided the specialist artist’s perspective. Lloyd Jones’ work 
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Fig. 3.2 Jessica Lloyd Jones ‘Forest Fuel’ 2018, permanent projection onto a biomass plant, 
Gothenburg, Sweden. (Copyright © J. Lloyd Jones 2018) included by permission 


explores concepts of energy, matter and natural phenomena through the interaction 
of materials and light through installations, projection and sculpture created from a 
variety of materials and processes. An example of this work is shown in Fig. 3.2, 
where a digital image is projected on to a biomass plant in Sweden. 

Producing works of this nature requires effective liaison with a wide group of 
people. These include project managers, funders and commissioners (Benefactors), 
technologists, engineers, architects, fabricators and design teams (Specialists) and 
curators (Translators). 

Internal communication between these people is, in part, influenced by the time 
sequence in which the various actors signed-up to the project. For example, com- 
missioners and project managers will often be in place before the artist becomes 
involved, whereas others (for example, fabricators) can only be selected by the artist 
once a project has commenced. The structure, roles and responsibilities of a team will 
often be assigned in line with the common goals identified between the artists, tech- 
nologists/scientists and the other stakeholders involved, with ultimate management 
responsibility depending on who instigated the project. Maquettes and visualizations 
provide useful aids here, illustrating ideas and concepts to stakeholders and potential 
stakeholders. 

External communication to an artwork’s intended audiences must simultaneously 
speak to many diverse individuals. Those with an active interest in the arts are the 
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easiest to reach as they tend to be more open-minded and curious. This is especially 
true when the information to be communicated is broad-based and thematic, for 
example, communicating ideas about energy, beauty, innovation or creativity. When 
audiences have little knowledge, understanding or exposure of the arts, this can 
directly impact upon enthusiasm, open-mindedness and curiosity, and subsequently 
on the success of the communication which follows. 

No one tool is the secret to successful communication, with the most effective 
means of communication depending on the context, though when engaging in a 
crowded market-place, novelty can be key. Audiences are more likely to be engaged 
if the artwork presents something they haven’t seen before. 


3.4.3 The Benefactor’s Perspective 


The benefactor’s perspective was offered by a senior manager in a UK-based public 
services innovation foundation, often delivering projects bringing together artists, 
technologists and public bodies. 

Audiences in communication in this environment are diverse. This is not only 
a diversity of organisations, but a diversity of individuals within each organisa- 
tion, which communication must be sensitive to. However, across all audiences, the 
communication of ‘facts and figures’ tends to be easier than the communication of 
‘concepts and ideas,’ thanks to: 


e The clear and unambiguous nature of the information: usually concerning who, 
what, when, where and how; 

e The clearly delineated audiences for the communication: with eligibility criteria 
defining the types of organisation for whom the information is relevant; 

e The simple and well-established channels through which communication is 
shared: for example, application forms to access funding and; 

e The simple feedback loops associated with these channels: for example, via 
workshops, emails and phone calls. 


Conversely, communicating concepts and ideas, especially the learning arising 
from projects, is generally more challenging. Points can be nuanced and subtle; the 
learning potentially applicable to any audience and the channels through which such 
learning is shared are less well-established resulting in feedback loops which can be 
vague or non-existent. 

Additional challenges can arise due to the differing organisational cultures in place 
between partners in a given project (especially those bringing academic, public and 
private sector partners together). Each may have different reasons for being involved 
in the project, and these need to be taken into account. At the same time, this diversity 
can be a strength of the project because it can lead to innovation and new ways of 
viewing the project and its outcomes, which can add value to the original concept. 

Exhibition is a tool used regularly to communicate concepts and ideas, present the 
outcomes achieved and learning gained through collaborative projects and to foster 


34 S. Liggett and M. Corcoran 


multiple conversations simultaneously between many partners. It is often easier to 
show someone something than tell them about it. The innovation foundation often 
use “pitching exhibitions’ to invited audiences of key stakeholders at targeted events, 
which they find to be a highly effective method. 


3.4.4 The Translator’s Perspective 


The translator’s perspective was offered by a public engagement manager at a lead- 
ing biomedical research facility. A permanent gallery at the facility, along with an 
accompanying public engagement outreach programme, aims to generate excitement 
and interest in discovery of science, human health and wellbeing: through artistic 
means and otherwise. 

Exhibitions have to cater to many audiences, marrying rich content for high 
science capital peers (including those researchers based at the facility) simulta- 
neously with that for a low science capital lay-public. Often, they also need to 
cater to visitors from different countries (with different cultural backgrounds), politi- 
cians, distinguished guests, as well as the accompanying press: simultaneously and 
seamlessly. 

To bring these exhibitions together requires close communication with and 
between: 


Public engagement and education teams (Translators) 
Press and marketing teams (Translators) 
The organisation’s scientists (Specialists): including those directly involved in 
each exhibition, and the wider community of those working in the organisation. 
Exhibition funders (Benefactors). 
Contractors (Specialists) of diverse backgrounds and specialisms. 
Focus groups (Publics and Beneficiaries) contributing to each exhibition’s 
relevance and accessibility. 

e Peers (Specialists and Translators): working in museums, galleries, and other 
science-communication environments, sharing best practice, knowledge and 
experience. 


In practice, the planning of each exhibition can require communication with a 
diverse group of 30—40 people. 

Internal communications are managed with a high degree of structure. An Internal 
Advisory Panel of senior managers and key stakeholders communicate the exhibi- 
tion’s broad themes and messages. It also translates these into the wider strategic 
direction and brand identity that the organisation is seeking to achieve. This helps to 
build trust—so that the senior managers and key stakeholders feel confident in the 
processes being followed and the expected outcomes. An exhibition steering com- 
mittee comprised of colleagues with a variety of roles across the organisation can 
provide feedback on various aspects of the exhibition plans. A project group is nor- 
mally required to discuss the practicalities of bringing each exhibition together and 
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meeting the needs of all stakeholders within the constraints of the space available. It 
is also important to maintain communication with any external teams of contractors 
and keep track of timelines, deliverables and issues as they arise. Communication 
within these structures is aided by video conferencing, formal presentations, docu- 
ment sharing and Gannt charts, but the most effective method is often to communicate 
face-to-face (or via video conference for more remote collaborators). 

Communication is most effective when it is between individuals who all work 
in the same context, with the same strategy, and the same audience. The greatest 
challenge is when the time is limited, audiences are disparate, and a contextual 
understanding is absent, communication is far more complex. 

With regard to the content of the exhibitions, to achieve their intended impact of 
making a tangible impact to the feelings and perceptions of their audiences, consid- 
eration needs to be given to the diverse ways in which they may be interpreted by 
different kinds of audiences, particularly where advanced or specialist subjects are 
involved. There is also a distinction between the explicit information the exhibition 
communicates directly and the implicit communication of its values, character and 
context. The latter can be assisted, amongst other things, by the atmosphere cre- 
ated by the exhibition and the personnel who bring it to life. Both are needed if the 
exhibition is to be successful. 


3.4.5 The Combined Perspective 


In many cases, one individual may take on multiple communication roles in their 
work. 

Such a combined perspective was offered by Jamie Harris, founder of Elder3D, a 
company specialising in making artworks affordable to the masses through utilization 
of cutting-edge 3D design and manufacturing technologies (Fig. 3.3). Harris is a 


Fig. 3.3 A metallic 3D Print (Left) and its supporting digital 3D Model (right). (Copyright © J. 
Harris 2019) included by permission 
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Specialist in both artistic and technological practices, a Benefactor commissioning 
work from his international network of associate artists and technologists and a 
Translator, ensuring the wants and needs of his customers are always catered to. 

In this context, understanding and trust are the basis for successful communication. 
Artists and 3D print specialists have to understand that the quality of the final product 
is dependent on the quality of information provided initially. The print specialists 
have to understand every detail in the brief and customers have to understand what 
is and what is not possible within the limitations of the technology. 

Communication is easiest when the information shared is quantitative and unam- 
biguous (for example, specifying the material and dimensions for a 3D print run) 
and when it is between actors who speak a common language, sharing common 
skills and experiences. Conversely, when the information to be shared is nuanced 
and qualitative (for example, when a family wishes to convey the ‘spirit’ of a person 
in a portrait) and when commonality of language is missing, communication is a 
challenge. Commonality lies in the shared experience of actors, but also on the num- 
ber of individuals comprising each ‘Information Source’. When a customer artist or 
printer is in practice, and when decisions are made by committee, information can 
quickly become confused, unclear or contradictory. 

Trust is a prerequisite to understanding allowing honest and open communication 
to take place customers must trust that their vision can translate into a high-quality 
product, and artists must trust that a printer can bring their designs to life, print 
specialists must trust that the designs will be suitable for 3D printing in the spec- 
ified materials and so on. Therefore, establishing a strong rapport and a personal 
relationship with everyone is essential to the business. 

With regards to exhibition, showing the actual work is often the best way to 
communicate because people can touch and handle the work. This creates excitement, 
promotes questions and produces a better understanding. 


3.4.6 An Interdisciplinary Communication Tool Wish List 


In Sect. 3.3 we proposed that the challenge of interdisciplinary communication was 
rooted in the many different ways and means required to communicate (impair- 
ing accuracy), the lack of common vocabulary and reference points amongst the 
principal actors communication is between (impairing precision), and the multiple, 
simultaneous objectives such communication is often required to serve (impairing 
impact). 

The views of those with direct experience of working in these environments cer- 
tainly do not contradict these conclusions. Rather, we observe many instances of 
these challenges in action. We also gain a deeper insight into how these challenges 
arise, the other challenges they are closely related to and how such challenges might 
be addressed. 
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Table 3.2 An interdisciplinary communication tool ‘wish list’ 


An interdisciplinary communication ‘wish list’ 


More accurate communication (to overcome technical challenges) 


1 Reduce the total number of channels needed to communicate messages 


2 Reduce the quantity and improve the quality of Transmitters/Receivers 


3 Reduce the risk of noise within multiple communication-feedback cycles 


More precise communication (to overcome semantic challenges) 


Bring clarity to open ended, subjective and qualitative messages 


5 Bring shared language, reference points and context to messages between diverse actors 


Bring clarity to single messages compiled by multiple actors 


More impactful communication (to overcome effectiveness challenges) 


7 Provide simultaneous communication of multiple (implicit and explicit) messages 


8 Build trust and rapport amongst the principle actors in communication 


Taking all of this into consideration, we propose a simple ‘wish list’ (see Table 3.2) 
of outcomes for any tool which could make interdisciplinary communication 
involving artists and technologists easier to facilitate. 

With a wish list for an interdisciplinary communication tool established, we can 
assess the extent to which the exhibition has the potential to be such a tool. This can 
be done by considering what exhibitions are and how well aligned (and potentially 
impactful) they are with respect to each of our desired outcomes. 


3.5 The Exhibition as an Interdisciplinary Communication 
Tool 


3.5.1 Defining Exhibition 


Exhibition is a broad and all-encompassing term. It may refer to any “event at which 
objects are shown to the public, a situation in which someone shows a particular 
skill or quality to the public, or the act of showing these things” [7], and it is this 
general sense that exhibitions are discussed here. This definition excludes private and 
informal day-to-day interactions. Furthermore, when we discuss exhibitions herein, 
we refer primarily refer to them in their most common temporary and ephemeral 
form, as opposed to permanent installations and collections. 

Our wish list (Table 2.2) is composed in the language of the Shannon—Weaver 
Model and so before we can assess exhibitions against each of its desired outcomes, 
we must first ensure we can describe exhibitions in a consistent way. For virtually 
any exhibition, in any form: 


e The ideas of the exhibiting artist and/or the curator serve as the Information 
Source. 
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The artworks, in their particular arrangement, are the Transmitters of these ideas. 
The exhibition environment is the Channel through which these ideas pass. 

The senses of the exhibition’s audiences are the Receivers of these ideas, and 

A given subset of the exhibition’s total audience is the intended Destination for 
these ideas. 


Furthermore, as done so throughout Sect. 3.4, the principal stakeholders in exhibi- 
tions can be simply mapped to the principal actors in interdisciplinary communication 
which we have defined as follows: 


Specialists: the exhibitors and/or the exhibition’s curator. 

Benefactors: the sponsors, institutions or others who fund the exhibition. 
Beneficiaries: the audiences who are the intended ‘Destination’ for the exhibition’s 
messages. 

Publics: formed from some combination of the above. 

Translators: curators and exhibition assistants. 


With these definitions in place, our desired outcomes can now be assessed in turn. 


3.5.2 Assessing Impact 


Exhibitions can address technical challenges, leading to improved accuracy in com- 
munication, by reducing the number of Transmitters, Receivers and Channels, needed 
to communicate information. Each Information Source has one Transmitter, an 
exhibit, fewer transmitters implies fewer receivers to get a message to its intended 
destination, and the exhibition provides the sole Channel for communication. Fur- 
thermore, its structured nature allows for multiple stable feedback cycles, provid- 
ing a constant background, providing a general context against which all further 
communications can be located. 

Exhibitions can address semantic challenges, leading to improved precision in 
communication, by front-loading the communication process with the information 
which does not require the use of technical and discipline-specific vocabulary and 
reference points to understand: that is, the general qualitative messages that lie behind 
the technical detail. The exhibits themselves become the common, visual and tactile 
transmitters used to communicate these messages, to actors who do not require 
any technical knowledge to receive them. By precisely communicating the general 
messages behind work at the earliest opportunity, subsequent discussion of that work 
(through symposia, informal meetings, exhibition guides or otherwise) and all that 
follows thereafter, becomes much easier, whether between one induvial or many. 

Exhibitions can address effectiveness challenges, leading to improved impact in 
communication, by simultaneously communicating to multiple and diverse actors. 
For some (for example, Publics), a general sense of ‘what the work is about’ is 
the desired outcome. For others (for example Benefactors), this general sense is a 
precursor to a desired outcome of a more detailed understanding why this work is 
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needed, at this time. For Specialists, this detailed understanding is also a precursor 
to understanding who they should work with, how they should work with them, and 
where they need to widen their technical understanding (or recruit the support of 
translators) to facilitate this. Each of these can use the same core set of transmitters, 
the exhibits, and the sole channel of the exhibition. The exhibition can be curated so 
as to allow multiple and mutually compatible routes through it. Some may view the 
exhibition online; others may view it in person. Some may use an exhibition guide; 
others take a guided tour. Some may participate in symposia; others use it as a venue 
for meetings. Some may spend only minutes in the space; others hours or days. All 
of this can happen at the same time. There is no direct analogy for this flexibility with 
a research paper, a seminar or many of the other traditional tools of interdisciplinary 
working. 

Not only that, but by putting an initial emphasis on common thoughts and feelings, 
in an environment which lends itself to being inclusive, social and highly creative, 
the exhibition plays a role in building the relationships, trust and rapport that are 
prerequisite to good communication, in a space which can entice and motivate new 
audiences to join in the fun. 


3.6 A New Curatorial Model 


The potential of the exhibition to provide solutions to interdisciplinary communica- 
tion challenges, as outlined in Sect. 3.5, is clearly encouraging, but our discussion 
so far is insufficient in and of itself. To support meaningful action and generate real 
outcomes, this potential must be pinned down and converted into the clear guidance 
required by exhibition curators and those responsible for making interdisciplinary 
communications work. Our observations need consolidating into a model, stipulat- 
ing what sort of exhibitions, will deliver what sort of outcomes, for what sort of 
audiences, and how. 

To date, there is no off-the-shelf conceptual model describing the functions of 
the exhibition as an interdisciplinary communication tool, and perhaps this should 
not be surprising. Exhibition and curatorial studies, and the discourse and theoretical 
models which underpin them, do not share a common genus with communication 
theory. They have evolved divergently, fuelled by different motivations, setting out 
to solve different problems and thinking about those problems in different ways. 

The Show-Talk-Do Model [4] (Fig. 3.4) describes the process by which high-level 
analysis of opportunities for research collaboration, alongside simultaneous non- 
expert engagement, can be achieved in environments which are rich with technical 
information and highly interdisciplinary. It contains three simple steps, which can 
be summarized as follows: 


1. Show It: Focuses on initial organic reactions. A public exhibition presents the 
organic products of the ongoing research of participating researchers, from across 
a variety of disciplines and interdisciplinary areas. Exhibits may include (but 
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THE SHOW-TALK-DO MODEL 
OUTPUTS 


HIGH-LEVEL ANALYSIS OF INTERDISCIPLINARY OPPORTUNITIES, 
ALONGSIDE PUBLIC AND STAKEHOLDER ENGAGEMENT 


INTERDISICPLINARY 


STEP 3 SCOPE FOR 
RESEARCH 
Bee DOIT COLLABORATION 
Ò ALL STEP 2 WIDERRESEARCH £ 
= RESEARCHERS TALK ABOUT IT CONTEXT < 
N Toven 
ND FEELIN 
PUBLIC SHOW IT AND FEELINGS 
INPUTS 


TECHNICAL AND DISCIPLINE SPECIFIC INFORMATION AND 
DIVERSE TECHNICAL AND NON-TECHNICAL MULTI-DISCIPLINE AUDIENCES 


Fig. 3.4 The Show-Talk-Do Model for multi-audience engagement (© M. Corcoran and Liggett. 
S. 2018) 


are not limited to) artworks, performances, documentation of practical research 
activities, visualization of ‘live’ research data and interactive demonstration of a 
process or procedure under development. 

2. Talk About It: Focuses on developing contextual understanding. A symposium 
(or similar) takes place, where exhibiting researchers discuss with one another 
the context in which their exhibit (and by proxy, their research) is situated. This 
discussion should be structured (in accordance with pre-defined rules) and be 
located within the exhibition itself. 

3. Do It: Focus on technical understanding and action. Research papers (or similar) 
are made available to researchers, and where synergy has been identified through 
Steps 1 and 2, the scope for research collaboration can be discussed. 


The exhibition itself lives within Step 1. Open to all (with Specialists alongside 
Benefactors, Beneficiaries and Publics in their various capacities), its primary role 
is to build common understanding on the level on thoughts and feelings, free from 
any pre-convinced ideas and prejudices: 


The exhibition should be curated as to allow each work to be considered in isolation, as well as 
all considered in combination ... Exhibit labels (if used) should not refer to exhibitors’ subject 
specific disciplines, their academic status or seniority, and should provide no information 
about the research presented. The exhibition should be made open and accessible to all the 
defined project’s stakeholders, including researchers, funders, policy makers, and the general 
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public. Visitors (including the exhibiting researchers themselves) should be encouraged to 
explore the exhibition with an open mind and reflect upon their spontaneous thoughts and 
feelings regarding the individual works and the combination [4]. 


When principal actors in communication have connected on an emotional level, 
they are then best disposed to discuss the wider context in which research is being 
undertaken (Step 2), and only after that, they are best disposed to engage in a labour 
intensive and technical discussion regarding the scope of future collaboration (Step 
3). These things can take place within the same exhibition environment, and over the 
same time period, but it is only when they are in this sequence, that the process will 
achieve its desired results. Also, different principal actors will be required to progress 
through different numbers of steps. For Beneficiaries, Step 1 may be enough, being 
inspired, challenged, provoked or otherwise by the works presented to them. For 
many Specialists, Step 2 will be enough, understanding their peers’ motivations and 
objectives, and only for those Specialists committed to working closely together (in 
virtue of what they have learned through Steps 1 and 2), are the challenges of getting 
to know the technical details of one another’s practice (Step 3) required. 

Show-Talk-Do was developed with a focus on interdisciplinary research. How- 
ever, this structure is applicable to communication in interdisciplinary contexts more 
broadly. It provides a simple conceptual model concerned with communication, 
considering the exhibition as a communication tool alongside all others, and pro- 
vides a helpful foundation for assessing the potential impact of the exhibition to 
bring about particular outcomes in particular circumstances, well aligned with the 
interdisciplinary communication challenges that have been identified. 


3.7 Conclusions 


This work has sought to establish what role, if any, the exhibition had in overcoming 
interdisciplinary communication challenges. 

The adoption of a conceptual model of communication identified that the chal- 
lenges that arise can be grouped in terms of accuracy, precision and impact. Through 
discussion with those with lived experience of these challenges, eight outcomes 
were identified which the ideal interdisciplinary communication tool would be able 
to tackle. The general ways in which the exhibition is well placed to meet these out- 
comes were identified. The adoption of the Show-Talk-Do Model enabled discussion 
on where these impacts might be most greatly felt, and how. 

Therefore, the exhibition, by virtue of its open, creative and engaging nature, its 
simple means of communicating messages and its situation at the nexus of many 
audiences, does have an important role to play in overcoming interdisciplinary 
communication challenges. It is likely to be especially significant when: 


e The actors engaged in communication are large in number and diverse in nature. 
e The messages being communicated are broad, qualitative and contingent on an 
“emotional/experiential’ understanding. 
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e The messages to be communicated are numerous and communicated to many 
actors simultaneously, each with their own objectives. 
Open and transparent communication amongst and between all actors is desirable. 
The communication process itself is intended to attract and grow its own audience. 


The exhibition does not replace other tools. On the contrary, It is most likely 
to have a major impact when delivered in conjunction with other tools, not only 
achieving its own ends, but serving as a means to foster higher quality discussions, 
and ongoing research and partnerships. 

However, if the exhibition is to consistently achieve these impacts and optimize 
its role as an interdisciplinary communication tool, there is still much work to do. 
We suggest the following three next steps are now required. 


(i) Better conceptual models 


The more mature the conceptual models, and the more articulate they can become 
at describing how exhibitions bring about their effects, the more useful they will 
become to the curators and other stakeholders in those exhibitions. Show-Talk-Do 
is a helpful start point, not an end. A sign of progress will be when it is rigorously 
scrutinized, refined and ultimately replaced, with better, more powerful models. 


(ii) Greater exhibition infrastructure 


It must become easier for those who want to use exhibition as a communication tool, 
to do so. Often, such tools are used not because they are the best option, but because 
they are the available option. Art and educational institutions, curators and research 
funders all have a responsibility to make exhibition an option: offering the expertise, 
guidance, facilities, resources and finance that are required to do exhibition well, and 
reap the just rewards. 


Gii) Leadership 


For exhibitions to be considered an essential communication tool, occupying the 
same status as a research paper or seminar in professional communication, it will 
require a shift in mind-set amongst the interdisciplinary community, and leaders 
from within that community to initiate change. Early adopters will face the biggest 
challenge, but each exhibition delivered will make the next one easier to deliver, and 
the impacts should follow closely behind. 

If these challenges are met, there is no reason why the exhibition should not be 
the mainstay of 21st communication practice in the age of the interdisciplinarian. 


Acknowledgements The authors wish to thank all those who shared their time and expertise to 
contribute to this chapter, most notably those we interviewed: Tal Stokes, Jessica Lloyd Jones, Rob 
Ashelford, Bryony Benge Abbott, and Jamie Harris. 


3 


Framing the Conversation: The Role of the Exhibition ... 43 


References 


D 


. Derrick, E., Falk-Krzesinski, H., Roberts, M.: Facilitating Interdisciplinary Research and 


Education: A Practical Guide. American Association for the Advancement of Science 
and the Colorado Initiative in Molecular Biology of the University of Colorado (2012) 
[online] Available at: http://researchdevelopment.byu.edu/wp-content/uploads/2012/12/Full_ 
Report_Creating_Interdisciplinary_Science.pdf. Accessed 23 June 2019 

National Academies.: Committee on Facilitating Interdisciplinary Research, Committee on Sci- 
ence, Engineering, and Public Policy. Facilitating Interdisciplinary Research. National Academy 
Press. p. 2 (2004) 


. Shannon, C.A.: Mathematical theory of communication. Bell Syst. Tech. J. 27 (July and October), 


379—423, pp. 623 656, (1948). Available at: http://math.harvard.edu/~ctm/home/text/others/ 
shannon/entropy/entropy.pdf. Accessed 5 March 2018 

Liggett, S., Corcoran, M.: Interdisciplinary research unmasked: a new curatorial model for 
multi-audience engagement. Stud. Vis. Arts Commun.: Int. J. 5(1) (2018) online ISSN 2393-1221 


. Derrick, E., Falk-Krzesinski, H., Roberts, M.: Facilitating interdisciplinary research and 


education: a practical guide. In: American Association for the Advancement of Science 
and the Colorado Initiative in Molecular Biology of the University of Colorado (2012). 
[online] Available at: http://researchdevelopment.byu.edu/wp-content/uploads/2012/12/Full_ 
Report_Creating_Interdisciplinary_Science.pdf. Accessed 23 June 2019 

Snow, C.P.: The Two Cultures, p. 3. Cambridge University Press, London (1959, 2001) 
Cambridge English Dictionary.: Exhibition (2019). [Online] Available at: https://dictionary. 
cambridge.org/dictionary/english/exhibition. Accessed 05 July 2019 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 


The images or other third party material in this chapter are included in the chapter’s Creative 


Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Chapter 4 A) 
Modern Communication Technologies get 
and the Marxist Understanding 

of Scientific Cognition 


Naira Danielyan 


Abstract The bicentenary of the birth of Karl Marx occurred recently (2018). It is 
thus apposite and insightful to reconsider his theories that relate to modern develop- 
ments in the field of communications technology. Marxism proposed the view that 
the basis of the human-specific attitude to the world is focused on practical activity 
directed to the transformation of natural and social objective reality. Scientific results 
focus on natural and social objects and processes and this blends into science-based 
technologies, interacting and being changed by humans who realize some objec- 
tive or social activity. Communication practice is connected with definite forms of 
worldview and a definite level of cognition, which reflects practical achievements 
and promotes their further reproduction and perfection. According to Marx, primacy 
of practice means that the efficacy of its real influence on nature, society, and per- 
sonality is a characteristic of the development of a subject’s cognitive activity. It is 
concluded that Marxism’s achievement is the development of the scheme: object— 
objective—practical activity—subject as the foundation of scientific cognition and 
the criterion for the correspondence between knowledge and reality. Nowadays this 
scheme is taken as primary while constructing and understanding communication 
technologies, considered as a special form of practice: this implies the dialectic unity 
of material artifacts on the one hand and human skills and experience on the other 
hand. 


Keywords Co-construction + Cognition - Communication technology • Marx - 
Marxism - Object - Practice - Subject + Technical activity + Trans-humanism 


4.1 Introduction 


Humans interact with the environment in varying degrees, ranging from passivity to 
aggressive pursuit of possibilities to transform it actively. Active transformation can 
be seen as being driven by the process of personal development as a social being 
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in the system of material and spiritual culture. The bicentenary of the birth of Karl 
Marx took place in 2018, and his ideas and theories still have plausibility, credibility, 
and relevance as tools to analyze and extrapolate from the situation today. Thus, it 
is relevant to consider their implications and application to the most transformative 
technologies of the present day. 

Marx considered man’s practical activity as the background of his specific attitude 
to the world (both natural and social). He wrote in “Economic and Philosophic 
Manuscripts of 1844” that man is not a product of circumstances and upbringing 
and we should not consider our mind as a passive sensation recipient. According to 
him, nature is a “man’s inorganic body with which he must remain in continuous 
intercourse if he is not to die” [1, p. 87]. While transforming the world, man creates 
both a new reality and new conditions of his existence that are not given to him in 
a ready-made form by nature. He also develops and perfects himself, specifically in 
his creative abilities. Hence, this results in the actual transformation of the physical 
world, which can be considered as the foundation of other manifestations of original 
human activity. Marx stresses that man is an “objective being” and “acts objectively” 
[1, pp. 92-94]. Further, he defines the way of including man in the natural world 
and society surrounding him through the active transformation of real objects and 
phenomena in the process of his praxis. This framework gives some initial guidelines 
to represent Marx’s interpretation of man in all the variety of his relations with the 
world as a whole. 

According to modern epistemological tendencies, the fundamental characteristic 
of practice as a specific output of man’s existence is its openness to objective reality, 
which always exceeds his abilities to master it. In contrast, there are still limitless 
human capabilities to develop new ways and means in order to interact with reality. 
As a result, the basis of the human attitude to the world is the creation of such a 
system that allows man to construct definite artificial renewable tools and means to 
influence his reality. He passes these from generation to generation; however, while 
creating, reproducing, and perfecting this “transformed nature” humanity transforms 
itself due to the formation and improvement of its appropriate skills and actions. 

Marx asserted that the realization of any forms of practical and transformative 
activity occurs only under conditions of social cooperation [1, p. 95]. It follows 
that people’s attitudes, as subjects of practical activity, to the objective reality that 
they transform always assume real human relationships as a concomitant. Thus, 
practice as a philosophical category is formed, reproduced, and developed in the unity 
of “subject—object” and “‘subject—subject’” relations: it is a manifestation of public 
activity. A human’s self-motivated intellectual evolution in practice is connected 
with the development of their communication skills and communication culture as 
necessary preconditions of mastering the external world. 

According to the theory of communication, this approach coincides with the infor- 
mation interpretation of the communication process, as can be found in the works 
of Shannon [2]. Its main characteristic is specific attention to the linear character 
of communication, i.e., it is an irreversible process. Following the communication 
paradigm, practices influencing the mass audience intentionally, e.g., mass media, 
government information, political structures or marketing, are dominant. They have 
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various forms of information-based influencing and techniques for manipulation of 
human consciousness in their foundations. 

Hence, communication practice, as with any others, will be always connected with 
definite forms of perception of the world and a definite level of cognition, focusing on 
practical achievements and promotion of their further reproduction and perfection. 
According to Marx, primacy of practice means that the efficacy of its influence on 
nature, society, and personality is a characteristic of the development of the subject’s 
cognitive activity [1]. 

Thus, practice as a specific form of human activity in the world possesses a 
complex system arrangement. It includes the following: 


(1) real transformation of the external environment by artificial tools and instru- 
ments (subject—object relations); 

(2) communication during this transformation (subject—subject relations); 

(3) a complex of norms and values (value and goal-oriented structures) that 
facilitates the targeted practical activity. 


4.2 Modern Epistemology About Subject—Object Relations 


Everything that really exists and is external to the subject, can be regarded as an 
object of cognition. It can be not only an item, but a whole complex or a system, 
either natural or ideal. 


Any physical thing existing in space and time, any objective and real situation can be con- 
sidered as an object of cognition. It can be the subject’s body, his/her consciousness or 
personality in general, another person, their consciousness, any cultural items (including 
books) and their meanings [3, p. 157]. 


The spectrum of the objects to be studied has significantly widened with time, 
due to an evolving system of ideals and norms of the cognition process, and this 
has allowed the discovery of ways to research and master complex, self-regulating 
systems. 

Special attention is currently being paid to discourse as a communication, i.e., ver- 
bal communication practice. Alexy [4] separates the following discourse rules, which 
he identifies as the most important points to understand the rational communication 
process: 


e Any subject who knows a language and is capable of functioning can participate 
in discourse; 
Any subject can question any assertion; 
Any subject can introduce any assertion into discourse; 
Any subject can express his/her aims, wishes, and necessities. 


Such an approach allows the achievement of a consensus due to the overcoming 
of personal subjective opinions by the participants of a discourse in favor of an 
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agreement motivated rationally. Both language and practices outside of language 
become important. They generate various forms of communication among discourse 
participants. 

As an example of substituting subject—object relationships by the process of obser- 
vation, it is useful to consider the ideas of the theory of autopoiesis by Maturana and 
Varela [5]. According to this theory, a human as “a living system” does not just reflect 
the world surrounding him/her, but constructs the worldview in correspondence with 
his/her cognitive, existential, and social sets [5]. As an autopoietic system, a human 
progresses independently while broadening his/her relations with the world. Accord- 
ingly, this leads to the appearance of inner and outer relations that then form his/her 
personal space. 

Autopoietic systems have the following features: 


(1) Operational insularity—the self-organization process is possible due to feed- 
back based on the system’s inner laws: it is not guided from the outside; 

(2) Informational insularity—any outer influence should be transformed into an 
inner cerebral activity because such a system can respond only to inner activity; 

(3) Physical openness—there is an exchange of substances and energy with the 
environment, and its character is defined by the autopoietic system. 


A system’s inner state forms the subject’s idea of the environment. However, there 
is always an outer observer who could play the role of the subject observed. In fact, 
this process of changing observers is limitless as there can be no last observer, i.e., 
an exclusive observation system does not exist. This means that absolute objective 
knowledge about the world cannot be achieved. Thus, it is possible to conclude that 
the achievement of some balance between reality and the result of a subject’s cogni- 
tive activity is impossible in practice: since a subject constructs his/her worldview, 
everyone has a personal reality. In this sense, a subject does not feel any bound- 
ary between their own experience and their conception of reality. Hence, cognition 
equates to adaptation: an object exhibits something that is correlated with the subject’s 
cognitive activity. 

According to the “observation of the second order” theory of Luhmann, observa- 
tion can consist of characteristics that create a complex system [6]. Cells, organisms, 
societies, and systems of artificial intellect are able to play the role of an observer. A 
second-order observation (or an observation of an observation) is different because 
of the circumstances of its observation, i.e., depending on whether an observer itself 
has performed it or someone or something has been observing him/her. According to 
the theory, an observer cannot see itself, i.e., it is not able to recognize itself during 
the observation. At the same time, however, another observer can see it. In this case, 
the object has the ability to see the observer and to see what is being observed. Luh- 
mann defines this process as a second-order observation (auto-reflection concept) 
and believes that it is these conditions that serve as the foundation of epistemology. 

The concept of observation allows us to avoid the traditional epistemological 
terminology that can be found in Aristotle’s works, in particular, the subject—object 
scheme. For example, it is possible to consider the relationships between a system 
and its environment while observing the system. “Countless discrete systems operate 
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inside a man as a condition of his life. They define the operations they implement by 
means of their own structures, despite being dependent on each other” [6, p. 11]. 
Thus, this explanation promotes the notion that the world of observation is created 
in the same way as any other system activities. 

Accordingly, the answers to the ontological questions: “what is it?” and “what is 
its state?” depend on a definite observer, who is restricted by the world of his/her 
observation. We can say that the observer is similar to some empirical entity that 
stipulates the selection of him/her as an object by other observers. It is obvious that 
the subject and the environment in which it acts are interconnected constructively 
and therefore experience mutual formation in the process of the subject’s activity. 

It is possible to conclude that nowadays cognition is considered as an active pro- 
cess of a subject’s activity. Firstly, cognition is the process of constructing the reality 
perceived by a subject through his/her cognitive operations, thus explaining why the 
knowledge that a subject receives, is a construct of the reality rather than of its reflec- 
tion. According to Apel [7], the subject of perception is now becoming involved in 
cognitive practice rather actively, as a result of interaction between perception means 
and objects on the basis of “significant thought’. This category requires a symbolic 
interaction, the nature of which is revealed most strongly in language [7]. There is 
such a statement in the works of Habermas, where he considers “communicative 
action”: according to him, this helps to overcome a number of antinomies, which 
have not been overcome by previous concepts of rationality [8]. 

Thus, the cognition process organizes a subject’s inner world model, but it does 
not solve the problem of how to describe the objective ontological reality. A subject 
is considered as a part of the world that s/he is going to observe or as a part of some 
system that is being observed by other observers. Thus, subject and object determine 
each other reciprocally. 

According to the interpretative constructivism theory of Lenk [9, 10], the inter- 
pretations connecting cognition and action run through all approaches to the analysis 
of the world and the subject of cognition: “Cognition is an action, but action is an 
interpretation... Hence, cognition is an interpretation. It means the process of cog- 
nition uses the results of the interpretation process or, to be exact, manifests itself in 
them” [9, p. 18]. Interpretation is not realized only by means of language as it is a 
more complex process: 


We think we might somehow grasp things immediately by language without taking into 
consideration that these matters are in reality much more complex. Language developed 
mainly as an instrument for preparing actions which we use in a similar vein like other 
means and instruments [10, p. 3]. 


According to these ideas, any view of reality depends on theoretical concepts, 
specifically the linguistic models that dominate in the society concerned. They 
include fundamental ideas, axioms, formal and linguistic instruments, and so on. 


The concepts of how we ‘grasp’ these ‘realities’ of different sorts and levels will be one 
main focus. ...methodologically speaking, we need a systematic approach to the processes 
of how the phenomena of knowledge and (re)cognition, perception, action and meaning can 
be analyzed from a systematic point of view [10, pp. 4-5]. 
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Hence, interpretation is an activity to create the so-called “mind constructs” which 
involve both ordinary and scientifically conceptualized structures. This approach 
allows the making of a model that contains only the aspects relevant to a subject’s 
purposes and its actions. The subject does not care about the model cognized: he/she 
thinks only of compensating the deviations while trying to achieve the appropriate 
purpose. 

As a result, any perception of reality will depend on theoretical concepts that 
prevail in the surrounding society, including fundamental ideas, axioms, formal and 
linguistic tools, etc. Hence, the preconditions to the formulation of modern ideas 
concerning the possibility of knowing the world are as follows: 


1. Whata person takes to be the experience of the world does not dictate the concepts 
which can explain the world. 

2. The concepts which can explain the world are products of historically situated 
interchanges among people. 

3. The degree to which a given form of understanding prevails or is sustained across 
time is dependent on the vicissitudes of social processes (e.g., communication, 
negotiation, conflict, rhetoric). 


According to this new approach, forces and relations of practice co-construct 
each other. This term, “co-construction’”, belongs to recent constructivist technology 
studies, which have rediscovered Marx’s idea of social and technical interdependence 
[11]. Contemporary technology studies offer concepts useful for analyzing these 
developments and also for identifying what they hide. In this domain, technology is 
conceived not as a pure product of inventive genius or an application of science, but as 
a “construction” of social actors. This leaves room for social choice between different 
designs that have overlapping functions, but better serve one or another social interest. 
This means that context is not merely external to technology, but actually penetrates 
its rationality, conveying social requirements into the human practical activity. 

According to Sukhodub [12]: 


Rationality in modern conditions serves as a method to develop critical thinking that is 
able to estimate crisis phenomena and to conceptualize the ways of overcoming them in 
social, economic, political, cultural and other spheres. It helps to make a conversion from 
authoritarian ideas that function formally in the society to the dialogic mind that is perceptive 
to ‘Others’ experience [12, p. 531]. 


Thus, it is possible to conclude that modern thinking culture is formed as a tran- 
sition from the postulating mind, which is based on some sustainable content of 
ideologies prevailing in the society, to the communicating mind, which is founded 
on the multiplicity of viewpoints and openness of the rational cognition system. 
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4.3 Practice as a Unification of Subjective and Objective 
Reality 


Practice is the foundation of all forms of human public activity. Its openness to 
the external world and its possibility to master new layers of objective reality in 
the process of practical transformative actions suppose an opportunity for regular 
development of a subject’s practical skills, e.g., his/her creative attitude to reality, or 
participation in various forms of social life. It leads to regular revision and perfection 
of the fundamental programs, which form the basis of practice. As a result, all the 
history of mankind, including material and spiritual culture, can be considered as 
a process of human creative approaches to nature in the form of new programs of 
practical activities. 

According to Hegel, subjective and objective reality contradict each other at the 
very beginning of the cognition process, as they constitute opposites. The life process 
starts with the individual necessity “to remove” objective reality that is opposed to 
it and, thus, “objectify” itself. This can be achieved through a subject’s “violence” 
to an object [13, p. 824]. It is possible to suppose that practice can be taken as a 
kind of objectification of a subject. Marx writes: “as objective activity everywhere in 
society becomes one of man’s essential efforts, all objects become his objectification, 
realization of his individuality, his objects” [1, p. 121]. Thus, practice is the unity 
of objective and subjective aspects of individual activity. It can be represented as a 
complex net of various acts of object transformations with the “objectification” of 
a human’s essence. Objects from one stage of this activity turn out to be the initial 
ones for another stage or another activity program. 

The structural characteristics of an elementary act of practice can be recognized if 
the labor process is taken as a pattern. Marx in his work “The Poverty of Philosophy” 
[14], discussing the cost of labor, reveals its following elements: man with his aims, 
knowledge, and skills; operations of his rational activity; and objects included in 
definite interactions in the course of these operations. Paying special attention to the 
labor object, he considers, that “labour is qualitatively defined by object and object 
in its turn is defined by specific labour qualities” [14, p. 32]. 

This pattern can be applied to the structure of practical activity and introduced 
as the unity of subjective (man with his abilities, aims, and rational actions) and 
objective (production means, initial materials, products of labor from initial mate- 
rials) aspects of the cognition process. There are various forms of human practical 
activity: according to Marx, the initial form of practice that lies in the foundation of 
other kinds and forms of human life activity, in general, is material production as a 
way of getting material welfare, which, in its turn, becomes the main driving force 
of social development. 


52 N. Danielyan 


4.4 Technical Activity as a Special Form of Practice 


Forming and developing public relationships is also a necessary form of practical 
transformative activity: it does not influence the environment surrounding a human, 
but it involves people themselves and their relationships. However, this form of prac- 
tice is initially connected with the material production practice. According to Marx, 
“the way of producing material life defines its social, political and spiritual processes 
in general” [15, p. 7]. Hence, this view proposes that there is a unified practical activ- 
ity that includes two aspects: man’s attitude to nature and his attitude to himself. It is 
worth noticing that during public interactions these aspects differentiate further and 
further since, due to the universal character of public activity, the subject can be both 
pieces of nature undergoing transformation and people whose “properties” change 
because of their inclusion in various social subsystems interacting in society as an 
integrated organism. Studying the first possibility, we can conclude that humankind 
deals with the human influence on nature, but the second postulated subject concerns 
the “subjective” character of practice as aimed at the transformation of social objects. 
From this point of view, humans can be considered as both the subject of practical 
activity and as its object. 

The early stages of social development witnessed the unification of subjective and 
objective aspects of practical activity. As the labor process became more complex, 
the operations that had been performed by man started to be materialized. They were 
carried out as a sequence of influences of one tool on another and after that on the 
object being transformed. Science as a special form of practice, having more narrow 
social significance than production or social instances, gained an increasing influence 
in society. It then turned into a direct production power and a means for control of 
social processes. 

Scientific cognition by itself possesses such qualities as a reflection of existing 
objects and these can be received or reproduced by different forms of practical 
activity. It also has projective and constructive functions, i.e., it provides knowledge 
about the objects being mastered during production and during social activity in the 
future. To check the validity of the knowledge requires a scientific experiment as a 
special form of practice: as a rule, this is typically based on production opportunities 
and human social experience within the current stage of development of society, but 
it overcomes the contemporary level of entities and foresees future technologies and 
ways of control of social life activity. 


4.5 Social and Cultural Perspectives of Communication 
Technologies 


In view of the accelerating pace of technological advancement, it is appropriate 
to consider ideas about the likely evolutionary paths of technologies that strongly 
influence society (especially communications technologies) before returning to 
consideration of the correlations with Marx’s concepts. 


4 Modern Communication Technologies ... 53 


At first, it is necessary to understand what the term “trans-science” means. It is 
often used by participants of international philosophical forums and conferences. 
According to Weinberg, the trans-science period started with the first A-bomb tests 
in the middle of the twentieth century. It stipulated the shift from researching nuclear 
power by some scientists to team projects: 


Many of the issues which arise in the course of the interaction between science or technology 
and society, for example, the deleterious side effects of technology, or the attempts to deal 
with social problems through the procedures of science — hang on the answers to questions 
which can be asked of science and yet which cannot be answered by science. I propose the 
term trans-scientific for these questions... [16, p. 209]. 


Biological effects of weak radioactive influence on the environment, consequences 
of applying nanotechnologies to prolong the life expectancy, emergence of vir- 
tual reality, creation of artificial intelligence, changes of human ethics under new 
technological impacts are among such questions. 

The main trans-scientific peculiarity is to unite problems from various fields of 
human knowledge, i.e., it is based on a trans-discipline approach. An outstanding 
current example is emerging and spreading of the Internet technologies. In compari- 
son with rather rational technological methods, used in the past, modern technologies 
are able to provide for numerous negative effects due to the availability of mecha- 
nisms possessing an opportunity to manipulate the human mentality. Moreover, there 
has been no comprehensive reflection of their properties so far. So, they might be 
accepted as the starting point of philosophical deliberations due to the fact that the 
traditional approach to understanding the technology cannot reflect all the range of 
arising challenges. As a result, the main question is how all the spectrum of tech- 
nologies and the social life are changed with penetration of the Internet technologies 
into absolutely all aspects of human existence. 

Scientific knowledge, as viewed from the position of trans-science, is understood 
as suggesting probable hypotheses, which pass through bifurcation points where the 
choice of trajectory for further development occurs. This subject of research particu- 
larly addresses complex, dynamic systems that include technical, managerial, social, 
and other levels. Any cognition process turns into a social act, since communica- 
tion among representatives of different knowledge fields stimulates the emergence 
of special norms and standards that are not connected with a specific author, but 
acknowledged as valid by all the scientific community involved in the process. As 
a result, they become a characteristic of a specific public style of thinking, with a 
consequential transformation of the communication language, which then acquires 
a universal character and influences linguistics and linguistic methods. These norms 
and standards permit cognizance of the complete range of science functions: 


To analyze languages of different knowledge fields. 
To discover “scientific discourse as a net of communications with their mutual 
intentionality and inter-reflection” [17, p. 496]. 

e To study natural sciences in the context of communicative relations: it is plausible 
that this view is also correct for social and humanitarian sciences. 
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However, these models have a number of drawbacks. Their acceleration, the exten- 
sion of their mobility and flexibility and their reduction of local attachment stimulate 
the arrival of location-independent micro-societies and new social institutions of 
global communications that can exchange information globally, e.g., in the form of 
text messages or, more powerfully, as social media. According to Nasarchuk, “man 
becomes a message generator” [18, p. 69], hence, a message performs a role as the 
fundamental item of the society, not a human. The whole set of messages, depend- 
ing on their intention and content, form the lifestyle of a person, a micro-society 
or a social institution. It does not matter who is a message carrier, because it is 
the content that gains a dominant significance. This means that its content and its 
author’s (supposed) competence are more important than its sender. Yet, a message 
and its perception are not always correlated with each other. The example of Internet 
technologies indicates a conclusion that a sender often knows nothing about his/her 
message recipient. In turn, the recipient is often not sure that the message has been 
sent to him/her personally. The recipient thus frequently cannot deduce what should 
be perceived as the message information as contextualized by some intention, i.e., the 
objectification of the discussion acquires some fuzziness. Antonovsky and Emelin 
suggest that “communication on the society level is still possible, but can’t form 
self-adjusting stable successions of messages” [19, p. 106], since the knowledge 
presented loses its discrete boundaries and metamorphoses regularly. 

According to the above conceptions, it is possible to highlight the following 
social and cultural perspectives of trans-humanism predictions under the influence 
of communication technologies: 


Forming a new lifestyle. 
Presenting a possibility of the phenomenon of “secularized eternity” in the public 
consciousness as a result of blurring the borders of reality. 

e Changing man’s life goals due to his rapidly evolving status as creator of natural 
and social worlds. 


Let us consider the final thesis in detail. An active role of cognition is the most 
important aspect of the application of these concepts to trans-humanism methodology 
[20]. It supposes the dominance of activity of the mind in all levels of perception, 
while nonstructured and nonclassified sensor data are absent. 

According to the ideas of trans-science [21], modern technologies will pro- 
vide new opportunities to create artificial intelligence and, as a result, to realize 
new social forms and psychosocial processes. The website of the Russian Trans- 
Humanism Society [22] cites some of its “radical possibilities” that could follow 
from the abovementioned trans-science issues and ways of their realization, such as 
the following: 


e Super intellectual machines. 

e Extension of life expectancy significantly. 

e Transfer of human consciousness into virtual reality (or brain downloading) and 
similar. 
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Trans-humanism manifests itself in three aspects: practical activity; technologi- 
cal achievements and social transformations. Trans-humanism can be considered as 
a synthesis of discrete doctrines being applied in technological practice, especially 
in the field of communication technologies. In this conception, a human being is 
substituted by a virtual existence, i.e., there is the potential elimination of anthro- 
pocentrism principles, which were laid in the foundations of traditional humanism 
and traditional science. 

Thus, the exploration of trans-humanism issues needs to be coordinated with the 
“stable development” conception [23], which can be understood as the necessity of 
the immediate protection of the environment, on the assumption of further conver- 
gence of natural and humanitarian sciences aimed at getting a more comprehensive 
understanding of a viable forward path. This approach raises a question concerning 
the boundaries of man’s constructive activity, its involvement, and correspondence to 
the real world. A human being is becoming more and more “technological” with the 
evolution of communication technologies; however, s/he doesn’t stop being sentient: 
the human, its body and consciousness turn into an integral part of complex eco-, 
sociocultural, and socio-technical systems [24]. 

In the context of the developing ecological and humanitarian crisis, the prob- 
lem of the human future can be considered as a task of preventing the degradation 
of human and natural systems, achieving co-evolution of science and society, and 
forming a civilization based on stable evolution by means of searching out sensible 
answers on topical matters: this correlates with a comprehensive worldview outwith 
trans-humanism. Communication technologies have a great potential to dominate 
these issues and define future perspectives of not only science, but the whole of the 
development of civilization. They can be both production tools and the final product 
of human fabrication activity. That is why they are becoming more and more impor- 
tant components of the production process, defining its character and content (e.g., 
“Industry 4.0” [25]). Communication technologies require a range of specific skills 
from their participants and as a result, imply the dialectic unity of material artifacts 
on the one hand and human skills and experience on the other hand. 


4.6 Extrapolation into the Future 


The development and introduction of new technologies have led to the emergence 
of a new sociocultural reality that raises new ethical issues. They are closely con- 
nected with the realization of various projects, e.g., complete description of think- 
ing processes and perception of reality by the human brain; the slowing down of 
aging processes; the opportunity of human organism rejuvenation; the develop- 
ment of brain/brain or brain/computer interfaces; the creation of robots and other 
devices possessing at least partial individuality; and similar speculations. Ethical 
problems arise from the realization of the above projects and the ethical principles that 
are conventionally followed nowadays will be transformed. Development and 
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penetration of these new technologies will provoke a cultural effect that is likely 
to see the intensification of some ethical values and the devaluation of others. 

For example, the future achievement of neurointerface technology would lead to 
the unification of human and machine on a qualitatively new level. It could change 
the level of virtualization of the human mind and social relations. Penetration of vir- 
tual technologies into human sensibility will create the situation of “hybrid reality”, 
which could obliterate distinctions between man’s virtual personality and his physi- 
cal localization in a body. However, the current virtual world of social networks, and 
especially social media, leads to egocentrism and the human’s preoccupation with 
itself and its thoughts: the result of this can be the loss of relationships between the 
human and reality. For this reason, discussion of change of the spatial conception 
concerning the physical boundaries of interpersonal communication and identifi- 
cation is very desirable. This change will involve a reconsideration of the human 
presence in the communication environment: whether it should be treated in both 
real and virtual forms simultaneously. Such an approach indicates a potential trend 
toward a completely new phenomenon of human existence: the boundary between 
real and virtual exists rather clearly nowadays. 

Thus, sociocultural perspectives of the current development of communication 
technologies include the following: 


The evolving appearance of a new lifestyle. 

Introducing a phenomenon of “secularized eternity” in public consciousness; 
Changing the meaning of human life in a substantial way as humans will be able 
to feel themselves to be creators of their world. 


Marx died in 1883. The telephone was invented in 1876, although the practical 
deployment of the telegraph dates from 1838. Thus Marx was alive at the beginning 
of near-instantaneous electrical communications, although the means of propagation 
of ideas most familiar to him would have been printed works: books, newspapers, and 
broadsheets. Extrapolation of his ideas to the circumstances of the present day must, 
therefore, be seen in this light. For him, the industry would have meant the making 
of hardware products and the concept of the “knowledge economy” would have been 
difficult to embrace: it is significantly different from material production as a way of 
getting material welfare (see Sect. 4.3, above). On the other hand, Marx’s assertion 
“the way of producing material life defines its social, political and spiritual processes 
in general” (Sect. 4.4, above) represents a more information-oriented view and his 
view that the realization of any forms of practical and transformative activity occurs 
only under conditions of social cooperation (Sect. 4.1, above) correlates closely with 
modern concepts of social media and co-production of media artifacts. 


4.7 Conclusions 


The philosophical exploration of social and cultural results of current technological 
development is becoming more and more topical. Today there is a real necessity to 
bring out the distinctive features of these technologies and to analyze their impact 
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on social reality. It is also very urgent to consider a new approach to humanism, 
as still understood traditionally, to clarify transformations of social values and the 
meaning of human life in the perspective of these developments, to study new cultural 
stereotypes emerging nowadays under the influence of communication technologies 
and emerging so-called “virtual reality”, social media, and artificial intelligence. 

Latour demonstrated the relation between technology and society [26] and the 
theme of democratization of technology has recently begun to receive the attention it 
deserves. As Latour has argued, the exclusion of technology from the social scientific 
concept of society is untenable. But once technology enters the picture, the issue 
of rationality appears in a new guise. It was Weber who introduced the concept 
of rationalization to explain many of the processes Marx had earlier identified as 
central to modernity [27]. Whereas in Marx’s works such processes were examined as 
potentially opposing (capitalist or socialist), Weber argued that they were the same for 
all modern societies. When there is a strong focus on technologies as there is today, a 
basis for questioning Weber’s simplification emerges again. It is clear that the modern 
world does not provide a unified view on the question of whether mankind is able 
to get a single homogeneous outcome to both technological and social development. 
The traditional practice activities as highlighted by Marx are blurring more and more 
as there is no sharp division between objective reality and the subjects of practical 
activity. The scheme “object— objective —practical activity—subject” as the foundation 
of scientific cognition and the criterion for the correspondence between knowledge 
and reality introduced by Marx has undergone a number of transformations in modern 
epistemology. Communication technologies with their new sociocultural reality erase 
differences between material and nonmaterial, nature and culture, theoretical and 
practical, etc. [26]. We do not need them anymore as the transition to “hybrid reality” 
and trans-humanism due to the equal interaction of living objects, social structures, 
theories or scientific-research programs, etc., generates a network consisting of a 
number of active elements. Its main characteristic is their collective actions. The 
coordination of their functioning in the network occurs due to the connections among 
elements, which makes all components interact. 

It is possible to conclude that there is a special meaning in these circumstances 
for such a form of practice as technical activity in the field of communication, and 
the analysis of Marx continues to give useful insights. This can be directed to the 
creation of not only material values and artificial environments, but also thinking, 
cultural, and cognitive processes. 
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Chapter 5 A) 
Use of Digital Holography to Re-Encode get 
and Image Chinese Movable Type 

Printing 


Shuo Wang, Ardeshir Osanlou, and Peter Excell 


Abstract In the long history of the development of civilization, Chinese movable 
type printing has played a pivotal role for disseminating information and it can be 
seen as the ancestor of the modern information age. From traditional engraving to 
type, from clay to lead type, from laser printing to 3D holographic printing, the tech- 
niques of encoding have been gradually developed and recorded, and these printed 
texts convey the progress of mankind. A brief introduction to encoding in the con- 
text of Chinese cultural communication is presented and, specifically, the role of 
encoding will be explored. In addition, digital holographic printing is introduced as 
a new approach and new medium to explore the enlightenment conferred by Chinese 
movable type printing on today’s society and especially the information revolution. 
Further, the future of encoding through the advanced technology of digital holo- 
graphic printing is discussed. Two holographic artworks are presented to illustrate 
the principles expounded and the response to their display in national museums 
discussed. 


Keywords Chinese movable type - Digital holography - Holographic printer - 
Printing technologies - Textual encoding 


5.1 Chinese Movable Type Printing 


5.1.1 Introduction to Chinese Movable Type Printing 


The invention of Chinese movable type is a remarkable milestone in the history of 
printing, as it ushered in a new printing era, well before the well-known developments 
in Europe. Chinese movable type printing has been used for nearly one thousand years 
and greatly promoted the development and exchange of world culture; in addition, 
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it contributed to the historical progression of world civilization [1]. Movable type 
involves making individual movable characters in advance, which can then be chosen 
according to the manuscript, arranged in line on plates and finally printed on paper. All 
of the movable characters may be used repeatedly, as each of them can be separated 
and rearranged. 

According to historical records from the ancient book MengxXiBiTan [1] (see 
Appendix), Chinese movable type originated in the Qingli period (1041—1048 CE), 
during the Song Dynasty, with clay movable type characters invented by Bi Sheng 
(Fig. 5.1) [2]. It should be noted that this was around 400 years before the introduction 
of movable type in Europe by Gutenberg [3]. Later, with the progress of technology, 
movable type characters made of wood, tin, lead, copper, and further derivatives 
appeared [4]. Figure 5.2 shows the Uighur wood movable type examples which were 
found in cave sediments of the northern Dunhuang area of China in 1908: these were 
the earliest surviving examples of wood movable type in the world [5]. 

The invention of Chinese movable type printing enabled the simplification of 
the process of printing plate creation, reducing labor requirements, and improving 
production efficiency. However, Chinese movable type printing did not immediately 
replace engraving-type printing until the Qing Dynasty, when the government began 
to promote text-based cultural transmission. After this, the number of movable type 
books greatly increased. Printing techniques also varied, as the Qing government 
directly organized human labor and material resources to support the development. 


ws 


Fig. 5.1 Replica of clay movable type characters based on Beisheng’s technology, in the Chinese 
Printing Museum 
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Fig. 5.2 Uighur wood movable type examples, as found in 1908 


For instance, during the Yongzheng period in the Qing Dynasty, copper movable type 
was used to complete as many as 160 million words on the integration of ancient 
and modern books [6] (Fig. 5.3). Further, in the Qianlong period, wooden movable 
type was used to print several thousand volumes of the Hall of Martial Valor book 
[6] (Fig. 5.4). Government participation was central in promoting the application of 
movable type and cultural communication with other countries. 

Today, movable type, embracing many of the ideas of the ancients, still has great 
value. Its most valuable point is its encapsulation of creative ideas, that is, the transfer 
of whole-plate engraving printing into individual units, whether phrases or characters. 
In today’s language, this may be called a code, symbol, or material carrier. This idea, 
in computer typesetting, is widely accepted as a general principle, which drives 
forward modern printing techniques [5]. Moreover, movable type printing is still 
popular among Chinese people. 


Fig. 5.3 The reference book—Qingding Ancient and Modern Integration 
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Fig. 5.4 The book—Hall of Martial Valor (Wuyingdian), one of the largest quantity of books 
produced by wooden movable type printing in China 


5.1.2 The Influence of Chinese Movable Type Technology 
in the Wider World 


Movable type was introduced into Korea not long after Bi Sheng invented it during the 
Song Dynasty. The original reason for using such an innovative printing technique in 
Korea was that Buddhist scriptures and Confucian classics imported from China were 
in short supply. In the beginning, Korea used ceramic movable type characters, but 
they were made in different sizes and this led to unsatisfactory printing quality until 
the Goryeo Dynasty, when copper movable type was gradually developed, which not 
only enhanced cultural identity between the two countries, but also promoted a new 
development of Asian culture [7]. 

After the invention of Chinese movable type printing, Japan adopted the tech- 
nique, mainly using wooden movable type printing characters to produce Buddhist 
scriptures for use among monks. A large number of Japanese books were printed, 
including the Japanese scripts modeled on Chinese cursive script [8]. Movable type 
printing was a major turning point during the process of social development as it 
improved the efficiency of mass book production and greatly promoted the dissem- 
ination of scientific, technical, and cultural knowledge. The spread of knowledge 
expanded from the aristocratic classes to the general population [8]. 

Vietnam commenced the use of movable type around 1841-1847 when the Viet- 
namese Dynasty bought a pair of Chinese wooden movable type sets and printed 
a large number of national legal books with it. This can be seen as the direct 
manifestation of Chinese type printing in Vietnam [7]. 

Besides Korea, Japan, and Vietnam, Chinese movable type printing also had a great 
influence on nations in other countries. In particular, Christian books in the Philip- 
pines were originally made by the ethnic Chinese people in local printing presses; 


5 Use of Digital Holography to Re-Encode and Image Chinese ... 65 


however, after 1608, the Filipinos set up their own printing houses. In addition, Iran 
was a transit station on trade routes, facilitating the introduction of China’s printing 
technology into Europe and hence making great contributions to the westward spread 
of the technology [6]. 

Europeans’ first contact with Chinese printing began in the Yuan Dynasty (1271- 
1368 CE), with the paper money brought by European businessmen and travelers 
returning from China [7]. Around 1450, A German inventor, Johannes Gutenberg, 
derived inspiration for a form of movable type printing from the technology of wine 
pressing machines [3]. Although this was more than four hundred years after Bi 
Sheng’s invention, the European movable type printing techniques introduced sig- 
nificant progress in the use of fatty ink, movable type made of lead, etc. Its mechanical 
printing technology laid the foundation for modern printing. These machines greatly 
increased the speed of production and promoted the formation and dissemination of 
new ideas: this became the driving force of modern civilization, and advanced the 
modernization of European society. 


5.2 Encoding in the Context of Chinese Cultural 
Communication 


5.2.1 Re-Definition of Encoding Concepts 


Encoding in semiotics can be interpreted as the process of creating a message trans- 
mission from an addresser to an addressee [9]; in psychology, encoding can be 
interpreted as the process of entering the memory system, storing, and subsequently 
retrieving [10]; in computers, encoding can be interpreted as the process of efficiently 
transferring or storing a sequence of characters (letters, numbers, punctuation, and 
symbols) into a professional format. In the book “Encoding and Decoding in the 
Context of Television Discourse,” by contemporary cultural researcher Stuart Hall, 
encoding is interpreted as meaning production of a media message and this plays 
an important part in cultural communication, and similar processes [11]. Based on 
these concepts, encoding is a specific process and has different definitions in different 
discipline areas. 

The present work attempted to identify the encoding of movable type printing 
in the context of Chinese cultural communication, hence encoding is defined for 
the present purpose as a transferring process from one role to another during the 
historical development of China’s movable type printing. In a physical view, movable 
type printing breaks the earlier whole-plate engraving printing into individual units 
and this may happen during the processes of deconstructing phrases or words. In the 
psychological view, movable type printing evolves from transmitter to protector of 
information (in cultural context), and from protector to inheritor. 
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5.2.2 The Role of Enlightenment 


The appearance of words can be regarded as the gestation phase of encoding. The 
invention of words was a great advancement for human civilization. In primitive ages 
in China, humans tied a number of different knots or different shapes of knots on 
ropes to express specific meanings: the so-called knot note (Fig. 5.5). Afterward, 
ancient humans portrayed certain things on stones and stone walls to convey certain 
meanings. This kind of pictogram directly gave birth to the origin of human writing: 
the archeological discoveries of ancient Egyptian hieroglyphic characters, Sumerian 
cuneiform characters, Chinese characters, and similar are all such. 

Chinese characters have a long history, but the exact time of their emergence is 
still a historical mystery [5]. Judging from the archeological findings, the Oracle 
script system was fully developed in the Shang Dynasty, around 3500 years ago. 
Chinese characters are ideograms whose form and meaning are closely connected. 
Early characters were of pictographic form: with such a glyph, people could easily 
know what it stood for. 

The usage of Chinese characters is closely linked with their transmission medium, 
but there were some limitations for expedient information transmission [12], for 
instance, scattered oracle bones were composed of animal bones with engraved sym- 
bols (Fig. 5.6); bronze inscriptions were heavy and difficult to move (Fig. 5.7); the 
words on bamboo slips were hard to modify (Fig. 5.8); silk as a writing substrate was 
expensive and susceptible to be eaten by insects (Fig. 5.9). The invention of paper was 
convenient for text recording, but it was not able to change the transmission mode 
of manual copying and thus still constrained the efficiency of transmission. Until 
the invention of whole-plate engraving printing, information could not be spread 
widely. Compared with manual copying, engraving printing had great advantages in 
saving time and improving efficiency, but it also had shortcomings: it was very costly 
as each version of the text required creation of a new set of plates. However, with 
the invention of movable type printing, whole-plate engraving printing was divided 
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Fig. 5.5 An example display of a knot note 
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Fig. 5.6 Oracle bone with 
inscription in the China 
Printing Museum (late 2nd 
millennium BCE) 


Fig. 5.7 Bronze tripod bowl 
cast for Duke Mao during the 
King Xuan period of the 
western Zhou Dynasty (828 
BC-782 BCE): the vessel 
has 497 characters in 32 
lines. (China Printing 
Museum display) 
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Fig. 5.9 Lao Zi silk manuscripts in the China Printing Museum, dated around 2000 years ago 


into separate re-usable units, overcoming many of the above problems and starting 
widespread information transmission. 


5.2.3. The Role of Encoding 


With the development of information dissemination, the role of encoding evolved 
from transmitter to protector then to inheritor, or in a global view it can be considered 
as a navigator, directed toward civilization. 

Whether as whole-plate engraving or movable type printing, the most obvious 
role at the beginning is as the information transmitter. In this process, information is 
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transferred from one form or format to another. It is also a medium for a communicator 
to translate its message or meaning into languages, images, symbols, and so on. In 
view of relationship between language and literacy, printing provides a platform for 
converting spoken language into written language, which promotes the development 
of language, especially the development of written language in the Chinese literary 
and grammar frameworks. Considering the relationship between written language 
and communication efficiency, movable type printing provides an acceleration effect, 
and a new language experience can be built on its platform. This process of evolution 
with experience is also a role of the transmitter. In other words, the transmitter 
influences the efficiency improvement between language and literacy. 

If the transmitter only transmits information and lacks cultural connotation, it 
cannot assist in advancing human civilization. Therefore, during the encoding pro- 
cess, the dominant role is held by the transmitter while the recessive role is that 
of the culture protector. In ancient times, the advancement derived from printing 
could be identified with history, religion, politics, military matters, and many other 
issues where printing had important historical and cultural values. The role of pro- 
tector is crucial in the transfer process, for instance there are still preserved the Vajra 
cchedika Prajfia Paramita (in English: The Diamond Sutra) (Fig. 5.10) based on 
engraving printing. This was made in the Tang Dynasty (868 CE), while the ency- 
clopedia MengXiBiTan [1] (see Appendix), based on movable type printing, was 
written in the Northern Song Dynasty (written c. 1086-1093 CE) [6]. There can be 
little doubt that the ancient imperial system also contributed in establishing special 
tules and orders for specific cultural preservation. 

Protection aims at re-dissemination and then assured inheritance. As the founda- 
tion of modern printing, movable type printing is the basis of pre-digital era printing 
technology (digital printing can be argued to be closely connected—see Sect. 3.1). 


Fig. 5.10 Vajra Cchedika Prajfia Paramita (The Diamond Sutra), printed in the Tang dynasty: this 
is a well-preserved piece with the name of the carving master and the year of carving (868 CE), it 
was discovered in the cave of Dunhuang in 1900. The picture on display is of a replica 
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Fig. 5.11 Brush pressing by a local craftsman onto wooden movable printing type in the Rui’an 
area of Zhejiang Province 


Here, another role of encoding is linked to deep ethnic cultural conventions. For 
instance, wood type printing was derived from the custom of renewing genealogy 
records, especially in the southern part of Zhejiang Province, China: the clan concept 
of root-seeking remains quite strong there. Every clan’s family tree is renewed every 
12 years or so. Figure 5.11 shows the use of wooden movable type printing in the 
Rui’an area of Zhejiang Province. In the wider view of communication theory, the 
expansion of social and cultural communication has promoted the popularization of 
education and the exchange of knowledge. Additionally, mass production of printed 
books increases the chances of book retention, reduces the possibility of handwritten 
copies becoming extinct due to limited collections, and has a huge positive influence 
on cultural heritage. 

Looking at the role of movable type printing in the history of civilization, it has 
always guided humans toward civilized societies. As a driver of social progress, 
movable type printing has effects on almost every aspect of modern civilization. Due 
to its roles as transmitter, protector, and inheritor, new ideas are quickly disseminated 
and popularized, and speculative thoughts are promoted. 


5.3 Digital Holographic Printing 


5.3.1 Modern Digital Printing 


The rise of computer-mediated printing has caused a radical change in printing tech- 
nologies, although this did not happen immediately. For many years the standard 
computer printers were the line printer and the electric typewriter, both of which 
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used metallic characters of a similar form to movable type, although not assembled 
in the same way as in a printing press: the daisy wheel printer can be said to be 
similar. Toward the end of the twentieth century, however, raster-based printers (e.g., 
laser and inkjet) became common and now totally dominate the market. These do 
not use hardware type characters in the same way as the earlier technologies, but the 
philosophical difference is not so great, since they store a vast range of characters 
in software, made available to the user as “fonts”, exactly mimicking the jargon of 
the traditional movable type printing technician. The fundamental point here is that 
human readers have become completely attuned to the use of discrete characters, as 
established by movable type, and the raster-based systems have to display text in the 
same way for it to be meaningful. 

A more radical difference with this technology is the ease of inclusion of 
illustrations, as these can be printed with exactly the same raster technology. 


5.3.2 Digital Holographic Printing 


Holograms can represent objects and scenes in three dimensions without the need for 
special viewing glasses. Computer-generated or computer-mediated holograms are 
intrinsically digital and hence also have a raster structure, but in three dimensions, 
hence the two-dimensional raster pixels are replaced by three-dimensional voxels 
in the software. However, the hologram is a two-dimensional object which contains 
the three-dimensional information. This means that a different implementation of 
raster pixels is required. Thus digital holographic printing is composed of a matrix 
of holographic pixels, also termed holopixels or hogels. Holopixels are created by the 
interference of three-color lasers in a holographic medium, such as a photopolymer 
or silver halide film. 

A digital printed hologram can visually represent an object in a realistic way and it 
also can be a creative medium to re-interpret the object in a narrative way. In addition, 
digital holograms can be duplicated repeatedly through a digital holographic printer. 

A digital holographic printer includes several components, such as the data source, 
tri-color lasers, a spatial light modulator (SLM), beam-steering optics, and a computer 
to show the images on the SLM and to control the whole system. The data sources 
should include a selection of resources, such as general artwork and illustrations, 
photographs taken by people, or digital imagery produced by software such as Maya, 
Sketch-up, 3D Studio Max, point clouds, and laser scans. The laser-writing scheme 
utilizes an object and reference beam pair. The object beam is modulated through 
the SLM using two-dimensional (2D) information synthesized from a scene. 

Digital holographic printing as an artistic medium has been practically developed 
and applied in cultural presentations. For example, Fig. 5.12 shows a holographic 
artwork created by author Shuo Wang: in this artwork, the digital printed hologram, 
as an innovative medium, not only enables the viewer to recover an archeological 
excavation space as it appeared in 1980, and to have a realistic spatial visualization 
of it, but also the losses of archeological detail owing to the difficulties of conditions 
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Fig. 5.12 Digital holographic artwork representing a reconstruction of an archeological scene, by 
Shuo Wang 


at that time could be bridged, bringing a profound influence upon archeological, 
historical, geological, and artistic representation of the priceless cultural relics that 
were recovered [12]. 


5.3.3 The Evolution of Chinese Movable Type Printing 
to Digital Holographic Printing 


Understanding the past is an essential component for projecting developments in the 
future. Chinese movable type printing has developed in the course of its historical 
changes. The process has been complex and diverse with both positive and nega- 
tive aspects, which are closely related to the social system, philosophies, aesthetics, 
and humanity of Chinese society in the context of historical development at the time 
[13]. With the initiation of the modern information age, the rise of digital holographic 
printing also needs to be explored in the context of today. Since the development 
of digital holographic printing and movable type printing occurred in very different 
historical periods, it is hard to compare them. However, if they are regarded as trans- 
mission media and linked in their aspect as information carriers with their respective 
medium characteristics, then they appear to have some similarities in particular peri- 
ods in their history. Thus, some comparisons and suggestions of similarities may be 
attempted: 
On the positive side: 


— 


They both are carriers of information, that is, media. 

2. Indifferent historical periods, both movable type printing and digital holographic 
printing have provided a new information dissemination platform for societies 
and have promoted language development globally. Compared to movable type 
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printing, digital holography carries more information or data and has the potential 
for the inclusion of highly realistic graphics. 

Both movable type printing and digital holographic printing have opened up anew 
mode of information transmission in their respective times. With the development 
of movable type printing, efficient replication using various materials has been 
created; digital holographic printing will also replicate with more materials in 
the future, not be limited to glass and photopolymer, but the digital generation of 
characters gives infinite scope to replicate not only the entire Chinese alphabet, 
but characters from other languages, mathematical symbols and new, fictitious, 
creations. 


On the negative side, neither of them has had a great speed of popularization. 


Chinese movable type printing was invented in the Song Dynasty but it was not widely 
used to replace engraving printing, which required intensive labor [4]. Currently, 
digital holographic printing is not widely used either. The main reasons are as follows: 


1. 


From the perspective of aesthetics, Chinese scholars advocated the beauty of 
carving, so, movable type printing was not rapidly developed after engraving 
printing [14]. Similarly, the digital printed hologram is not at present as realistic 
as the traditional analogue hologram, and it cannot provide the same beauty and 
verisimilitude. This deficiency is expected to disappear as holopixel resolution 
increases. 

From the perspective of the development period, after the invention of engraving 
printing, people constantly improved its technology. In the Song Dynasty, it 
was the golden era of engraving printing when Bi Sheng invented movable type 
printing [1]. Digital holographic printing also seems to have its rival technologies, 
such as head-mounted virtual reality displays, but the usage scenarios for these 
are very different. 

From the perspective of economics, due to the features of Chinese characters, 
it is difficult for ordinary people to invest in the necessary labor and material 
resources (excepting the very wealthy) as it is very costly for one-time use. 
Before the emergence of movable type printing, there were numerous engraving 
workers. Because of that, the cost of engraving printing was lowered, but the 
cost of movable type printing was effectively increased. At present, it costs a 
great deal in time and money to produce a digital printed hologram, and digital 
holographic printers are at present very rare. 

From the perspective of media-use habits, after movable type printing was 
invented, handwritten and engraved books were still generally used at that time. 
Now digital color printers and 3D display monitors are also competitors for 
digital holographic printing. 
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5.4 The Exploration of Encoding in the Future 


The encoding of digital holographic printing can be seen as paralleled by movable 
type printing in some cases. In view of the medium features and advanced appli- 
cations, digital holographic printing will also experience the roles of transmitter, 
protector, and inheritor. However, due to the rapid growth of information and the 
diversification of information media, digital holographic encoding ideally needs to 
be combined with artificial intelligence, so as to optimize it. For example, throughout 
the whole process of digital holographic modeling, this process can be performed 
with a large number of databases based on people’s creative ideas, functioning as 
a replacement of manual modeling, improving the models, material, lighting, ren- 
dering and other steps, and designing the frame narrative through smart thinking. 
During the printing process, smart devices can hasten transmission speed, shorten 
printing time, and achieve efficient replication. 


5.5 Artistic Works 


5.5.1 Three Manifestations of a Chinese Movable Type Text 


Based on the research on encoding of movable type printing, as well as the relation- 
ship between movable type printing and holographic printing from the perspective of 
media analysis, an artwork entitled “CUBES” was created to further explore the dif- 
ferent roles in the future. The name “CUBES” involves two levels of significance: the 
straightforward meaning is descriptive of the artwork represented by different pieces 
in the form of small cubes; secondly, CUBES is an acronym, meaning “CUlture 
Broadly Encoding Spreading”. The work was reproduced in the form of collocated 
wooden movable type, 3D printed movable type and holographic movable type, and 
the record of movable type printing in MengXiBiTan [1] was taken as the con- 
tent (see Appendix). This follows the philosophy of the four “RE” creation methods 
(Re-definition, Re-creation, Re-construction, and Re-presentation) devized by author 
Shuo Wang [12], as well as holographic montage ideas. Each individual text com- 
ponent has been created in the 3D software: initially, the length, width, and height 
of the components have been calculated based on the wooden texts; furthermore, the 
polygonal modeling method follows the style of Chinese wooden engraving and the 
corresponding writing style, Fig. 5.13 shows holographic modeling of the content of 
MengXiBiTan and Fig. 5.14 shows the holographic rendering result, which simulates 
the effect of Chinese wooden engraving. 
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Fig. 5.13 The “CUBES” artwork: holographic modeling in Maya 
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Fig. 5.14 The “CUBES” artwork: rendering effect in Maya 


5.5.2 Four Manifestations of Chinese Movable Type 
in a Hybrid Timeline 


This artwork (Figs. 5.15 and 5.16) explores the different roles in the context of 
encoding. In addition, digital holographic printing was used as a new approach and 
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Fig. 5.15 Artwork: Hybrid Plate as a timeline 


past information disseminating 
future 


Fig. 5.16 Interpretation of this artwork 
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Transmitter 


Protector Inheritor 


Presenter Receiver 


Fig. 5.17 The psychological and philosophical view of encoding 


new medium to explore the enlightenment conferred by Chinese movable type print- 
ing on today’s information revolution. This artwork consisted of a wooden plate and 
266 individual blocks, including wooden movable type blocks, analogue holographic 
blocks, digital holographic blocks, and black blocks, these four parts (Fig. 5.2) briefly 
present a timeline of the information dissemination from ancient times to the future. 

In creating this work, the author Wang assumed two roles/identities in the 
creation process: information receiver and information presenter. Furthermore, he 
attempted to explore the relationships among transmitter, receiver, presenter, protec- 
tor, and inheritor to address the psychological and philosophical views of encoding 
(Fig. 5.17). 

Considered as a receiver, the information received was drawn from the content 
of the “Ancient Chinese Wikipedia” —MengXiBiTan [1], In the process of receiving 
the information, the whole content has been summarized into 70 characters, and the 
these characters have been engraved into wooden blocks, based on the traditional 
method of Chinese wooden engraving: this aims to present the ancient method of 
information transmission for audiences. 

Considered as a presenter, two approaches have been explored, based on the 
medium characteristics of analogue holography and digital printing holography. Ini- 
tially, the 70 characters were produced using a Holo-Camera (Fig. 5.18): these holo- 
graphic blocks can be seen as one of the modes of protection for wooden blocks (via 
substitution, since the 70 wooden blocks are considered as heritage objects); addi- 
tionally, they can encapsulate the moment of creation of the wooden blocks, which 
provides a spatial view for audiences. 

As a further insight, in the process of the digital holographic modeling, other 
spatial aspects of the characters can be discerned, as well as the opportunity to re- 
interpret the character in a narrative way. In this artwork, when each character was 
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Fig. 5.18 Using a Holo-Camera to produce the analogue hologram 


in the processing of being created, the structure revealed a layer of new meaning and 
spirit. 

In such an era, facilitating the production of the hybrid printing plates, each 
individual block carries different dimensions of time and space, from the ancient 
wooden type block to the latest digital holographic blocks. Thus technology, medium, 
and culture have been developing together: they act the different roles in the process 
of information dissemination. Hence the development of printing technology and 
culture has always been developing in a synergistic way. 


5.6 Conclusions 


The historical development of Chinese movable type printing has been reviewed 
and its influence on the World’s culture emphasized. The encoding of movable type 
printing has been explored and this expanded to include the evolution of Chinese 
movable type printing to digital holographic printing, including reinterpretations 
of the concept of encoding. Art works have been presented that demonstrate the 
coexistence of ancient movable type printing and modern digital holographic printing 
and the insights that can be drawn from this. The power of the holographic image has 
been demonstrated since, when the artwork was illuminated by the appropriate light 
source, virtual (holographic) character cubes, and real character cubes were combined 
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together, thus presenting a new form of encoded space within the culture of human 
communication. In addition, the artwork reflects the passage of thousands of years of 
historical space and time, linked through the continuity of Chinese characters. The 
re-encoding of Chinese movable type printing reveals that this printing technology 
continues to develop from the past to the future—in digital holographic forms—with 
the potential to inform new interpretations of human communication. 


Appendix: Records of Movable Type Printing 
in MengXiBiTan 


MengXiBiTan is a book written by Shen Kuo, a scientist in the Northern Song 
Dynasty. It was written around 1086-1093 (CE) and documented Shen Kuo’s views 
and opinions in his life. It is regarded as an Encyclopedia of ancient China by western 
scholars, with many foreign language versions published around the world. 

In the Tang Dynasty, people had not yet adopted engraving printing to print books 
on a large scale. In the Five Dynasties (907—960 CE), namely, the Later Liang Dynasty 
(907-923), the Later Tang Dynasty (923—936), the Later Jin Dynasty (936—946), the 
Later Han Dynasty (947-950), and the Later Zhou Dynasty (951-960), Feng Dao 
began to print the Five Classics with the engraving printing technique. After that, 
various classics and books were printed. During the Qingli period, a civilian named 
Bi Sheng created movable type printing. His method was to use clay to engrave 
characters and the thickness of each word was like the edge of copper coins. Each 
character was a matrix, hardened by firing. Firstly, he set up a piece of iron plate, 
covering it with turpentine and wax mixed with burnt paper ash. When printing, he 
put an iron frame on the iron plate, then arranged the word (i.e., character) matrix 
appropriately and closely. He filled the iron frame as one printing plate, then held it 
near the fire to bake; when the rosin and other bonding materials began to melt, he 
pressed the surface with a flat plate. In this way, the pattern on the plate was flat and 
smooth. If people wanted to print only two or three books, this method would not be 
convenient; but this method would be particularly efficient for tens or even hundreds 
of books. Usually two iron plates were produced while printing. One plate was used 
for printing and the other one for arranging the matrix. As soon as one plate finished 
printing, the other one would be ready to use. The process was quite fast with two 
plates. Each word had several matrix elements. For example, the Chinese words “ 
Z” and “t” usually had more than twenty matrix elements in case of repetitive 
characters in one plate. When the words were not used, they would be marked by 
pronunciations with paper strips. Each rhyme part was marked with a label, then 
stored in a wooden lattice. If an unfamiliar character was needed but unprepared, 
people engraved it and then grilled it with a grass fire, hence it could be made very 
quickly. The reason why people did not use wood to make movable type matrices 
was that the texture of the wood varied between sparse and dense. Once stained with 
water, it would expand, and easily got stuck to the frame, making it hard to remove. 
Thus, clay was a better choice. After the matrix was used, the coating material could 
be melted by fire. Wiped by hand, the matrix would fall off and not be contaminated 


80 S. Wang et al. 


by coating material at all. Bi Sheng’s matrices were understandably still treasured 
by his cousins and nephews after his death. 


References 


1. Shi, Jinbo.: The history of movable type (in Chinese). J. Guyuan Teachers College (Soc. Sci.), 
39—40 (2001) 

2. Hao, Liu.: Movable Type: The Rebirth of Wood Movable Type—The development and Inno- 
vation of Wood Movable Type Under the Modern Aesthetic Demand. Design (J. China Ind. 
Des. Assoc. Beijing). ISSN 1003-0069. (2016) 

3. McLuhan, M.: The Gutenberg Galaxy: The Making of Typographic Man (1st ed.). University 
of Toronto Press (1962). ISBN 978-0-8020-6041-9 

4. Zheng, Yefu.: The origin of movable type printing (in Chinese). Social Sciences of Beijing. 
ISSN 1002-2054. (2015) 

5. Shi, Jinbo.: The history of movable type (in Chinese). J. Guyuan Teachers College (Soc. Sci.), 
41 (2001) 

6. Shi, Jinbo.: The earliest extant printing products in the world (in Chinese). J. Beijing Lib. 
(1997) 

7. Wang, Xiaoxue.: Review and Reflection on the research of Gutenberg’s metal movable type 
printing (in Chinese). Research on the History of Publishing in China, pp. 59-67 (2016) 

8. Xu, Yinong.: The Significance of East Asia Typography in World History (in Chinese). Nanjing: 
New Century Library (2016) 

9. Encoding (semiotics). https://en.wikipedia.org/wiki/Encoding_ (semiotics). Last accessed 14 
March 2018 

10. Encoding. https://www.alleydog.com/glossary/definition.php?term=Encoding. Last accessed 
14 March 2018 

11. Encoding and Decoding in the Television Discourse: Centre for Cultural Studies. University 
of Birmingham, UK (1973) 

12. Wang, S., Osanlou, A., Excell, P.S.: Case Study: Digital Holography as a Creative Medium 
to Display and Reinterpret Museum Artifacts, Applied to Chinese Porcelain Masterpieces, 
Chapter 6. In: Earnshaw, R.A. (eds.) Research and Development in the Academy, Creative 
Industries and Applications, pp. 65-88. Springer, Cham, Switzerland (2017), ISBN: 978-3- 
319-54080-1 

13. Li, Bin.: Looking for the living fossils of movable type printing. Human. Photogr. 5—10 (2016) 

14. Zhu, Qin.: Discussion On the Reason of the Slow Development of Ancient China Printing 
(in Chinese). China Publishing Update (China News Publishing & Media Group, Beijing), 
pp. 40-42 (2002) 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Part Hl 
Interactions Between the Arts and Data 


Chapter 6 R) 
The FOREVER-DO Game: A Big Data geai 
Fishing Expedition 


Jill Townsley and Carlo Ferigato 


Abstract The FOREVER-DO Game is a participatory visual artwork, that takes 
the form of a relational game and installation. It is one outcome from an art science 
collaboration supported by the European Commission’s Joint Research Centre in 
Italy. The artwork was developed under the theme of ‘Big Data and Identity’ it 
considers the causal chain that our individual and social actions may have as they 
flow across the interface of the human and the digital network. The work focuses on 
data flow that has some correspondence with coordination and communication, as 
identified by Petri Nets theory. Carl Adam Petri introduced the Nets in the 1970s, 
they are one of the techniques used today for the analysis of Process Data. When 
data is ordered in time, the causal chains of data are collected in ‘processes’. One 
simple representation of a process net is the ‘bucket chain’ used to represent two 
fundamental aspects of data flow: selection and transfer. By using the bucket chain 
as a platform this project is focussed on a discipline of data where some ‘causal 
flow’, possibly represented by nets, has a role in the data analysis game. The resulting 
artwork encourages the viewer/participant to flow through an installation following 
a set of rules on a box, their actions coordinate according to the incidence of chance, 
selection, transfer and the similarity or difference of everyday objects. 
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6.1 Introduction 


Artistic activity is a game, whose forms, patterns and functions develop and evolve according 
to periods and social contexts; it is not an immutable essence [1]. 


The FOREVER-DO Game is a socially relational public artwork or happening, in 
which people flow around a space, moving between a network of coloured doormats, 
according to a set of instructions printed on a box. This seemingly absurd game 
activates interactions between players, who meet randomly in space and time. The 
coordinated nature of the rules-based system dictates the path that the players travel, 
it is in itself a socially related construct. A construct that rationalises data flow, mak- 
ing visible the hidden nature of all data flowing within our big data systems. The 
work presents us with a symbiotic relationship between scientific theory and art prac- 
tice, while also raising questions about socially mediated, rules-based interactions, 
especially those that cross the physical and digital interface. 

The game is positioned within today’s digitally networked age, it mirrors the 
theoretical construct of a ‘Bucket Chain’ network, a simple form of Petri Net system, 
proposed by the computer scientist Carl Adam Petri in the 1970s. In theoretical 
Computer Science, nets are instruments used for the analysis and design of systems, 
distributed in time and space. The strength of these nets is their explicit representation 
of fundamental situations of coordination and concurrency among system agents. 
Agents can be anything: objects, computers and/or human beings. 

The first manifestation of this work was played/exhibited during the Milan Digital 
Week in March 2019, in the Palazzo dei Giureconsulti, near the Duomo, in collabo- 
ration with the MC3 research group on concurrent systems at the University of Milan 
Bicocca. The game was repeated in October 2019 for the Resonance III festival, at the 
European Commission’s Joint Research Centre (JRC) in Italy, this time with addi- 
tional digital visualisations developed through the application of Radio Frequency 
Identification tracking (RFID). Trackers were placed inside each of the 2,000 boxes 
used to play the game, the feedback from the trackers was detected by antennas with 
the data then displayed digitally alongside the installation (Fig. 6.1). It is the first 
presentation of the FOREVER-DO Game in Milan that we discuss here. 

The cross-disciplinary collaboration between artist Jill Townsley and computer 
scientist Carlo Ferigato was important to the way that the artwork has developed. The 
work was commissioned by SciArt, an art science organisation embedded within the 
JRC. The JRC is the European Commission’s science and knowledge service which 
employs scientists to carry out research in order to provide independent scientific 
advice and support to EU policy. The SciArt mission: 


brings together scientists, artists and policy makers to discuss matters of concern from various 
points of view, not only to the JRC and the European Commission but also more widely to 
society [2]. 


Additionally, SciArt set the agenda of “big data and identity’ as the theme for the 
collaboration, stating: 
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Fig. 6.1 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019 


Big Data have erupted in our daily lives and we want to join the discussion on how to turn 
Big Data into culture, not leaving it only in the hands of technicians, trolls and corporations. 
We want to become hackers of our own future. We think it is time we make them ours, these 
Big Data, this digital transformation, in a vast collective effort open to all [2]. 


To do this SciArt have proposed a new concept ‘DATAMI’: 


We see the datami as a virtual tatami [a Japanese mat], made of the data that we cherish, our 
data and those of our families and friends, our discoveries and curiosities, our roaming and 
conversations, all of these making up our new identities [3]. 


This is the theme for the Resonances III Festival and Exhibition held at the JRC 
in October 2019, and it is within this overarching context that the FOREVER-DO 
project was formed (Fig. 6.2). 

In order to help convey the significance of the FOREVER-DO Game we look to the 
work of art critic and curator Nicolas Bourriaud and his concept of relational aesthet- 
ics. This supports the identification of some formal qualities in the artwork especially 
the interpersonal cooperation that is present within the FOREVER-DO Game and 
the significance of this model of interaction as a creative tool, to critically reflect on 
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Fig. 6.2 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019 


society. We also look to the media theorist, critic and philosopher Boris Groys and 
his writing on ‘flow’. The flow of people and objects within the game is important 
and Groys helps identify how flow may define moments of temporary happening. 
Happenings that can have some governance over social interaction, presenting as ‘an 
event not as a thing’ [4]. Groys’s theory helps us draw some conclusions about the 
way that the game mediates the viewer/participant in order to highlight the interface 
of the digital and physical realm, especially in relation to big data operations and to 
net theory. 

Finally, to help us consider how the artworks analogue process interacts with the 
human digital interface we will be looking not only to Petri Nets theory, (as the 
inspiration for the way that the game has been developed), but also reading Carl 
Adam Petri’s writing as a critical tool. Looking specifically at his Communication 
Disciplines delivered as a set of twelve conceptual tools. He proposed them in a 
lecture in 1976 to support reasoning in communication systems design, and sug- 
gested that they could help build the ideal computer system [5]. A system that could 
enhance human cooperation and overcome some of the many problems he foresaw 
for a computerised world going forward. Petri’s principal aim was to help realise 
a computer system whose applications could support humanity to develop its best 
possible future. It is this call from the 1970s that dovetails so significantly with the 
concept of DATAMI proposed by the SciArt team and motivates the creation of the 
FOREVER-DO project. 
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6.2 A Fishing Expedition—the Method 


Meeting within the context of the JRC, and in response to the SciArt theme of 
Big Data and identity, we (artist Jill Townsley and computer scientist/researcher 
Carlo Ferigato) realised that ‘process’ held an important position within both of our 
working practices. Ferigato’s research around the process design of communication 
systems, specifically systems identified by Petri Nets theory, and Townsley’s artwork 
emphasising the process of the art object, definitive beyond subject and object. The 
process presented us with common ground to articulate our practices in language that 
cut across our respective fields, producing a rich dialogue and exchange of creative 
thinking. It identified important key words beyond the process itself, such as system, 
time, space, flow, coordinated behaviour, repetition, difference, unfolding, folding, 
selection and transfer. 

Through analysing our processes, nets became a key link between the art and 
science used in our collaboration. Put poetically, we became fishermen casting our 
nets, to fish across disciplines into the sea of big data sets in order to extract coher- 
ent patterns and procedures. We found patterns pertinent to Petri Nets theory, that 
could be visualised as art. From this, we have developed two symbiotic artworks the 
FOREVER-DO Game and the FOREVER-DO Infestation. 

While we consider only the FOREVER-DO Game here, it is important to know 
that the placement of boxes within the game became data that has driven another 
artwork, the FOREVER-DO Infestation. The data, gathered from one piece of work 
flows on to inform the realisation of another. In this way, authorship is shared with 
every person who has played the game by placing a box on a pile. 

While the FOREVER-DO Project as a whole (the Game and Infestation) explores 
the idea of ‘fishing’ into data sets, the nets we use are multifariously allegorical. 
Beyond being a fisherman’s tool, the word ‘net’ has reference to our digital networks, 
the Internet. Nets are also the title given to the formal computer science tools ‘Petri 
Nets’ (as outlined in the introduction). One of Petri’s examples, the “Bucket Chain’ 
is the main source of inspiration for the FOREVER-DO Project. It is outlined in 
some detail in the theoretical context below but is an explanation of the mechanics 
of the FOREVER-DO Game. 


6.3 The FOREVER-DO Game—the First ‘Catch’ 


This artwork propels its human participants on a physical journey that mirrors a 
‘causal flow’ of data (Fig. 6.3). The basic idea of causal flow is to understand how 
factors influence one another. The game seeks to envision the invisible influence 
between human action and data. As the game is played, a sculptural installation con- 
sisting of towers of boxes develops in the gallery space. These box-towers simulate 
a data pile, presenting a physical object that records the coordinated meetings of 
each individual who (following a set of instructions on the box) encounters another 
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Fig. 6.3 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019, 
included by permission 


player at a coloured mat and finds (through random chance) that their boxes contain 
identical objects. 


The mechanics of the game are as follows. In a large room there are placed a set 


of objects: 


Sixteen coloured doormats, placed randomly around the room, specifically, 


— 4 Yellow, 

— 4Red, 

— 4Blue and 

— 4 Black doormats. 


2,000 boxes (made to hold a ream of A4 paper). 


— 1,000 brown boxes—randomly dispersed through the installation, 
— 1,000 white boxes—randomly dispersed through the installation. 


Each box has instructions (the rules of the game) printed onto the top. 


— There are 6 sets of rules offering slight variations. 
— The 6 sets of rules are randomly distributed (shuffled) throughout the 2000 
boxes. 


Each box contains (randomly) one item of mass-produced wooden cutlery: 


— aknife, 
— fork or, 
— spoon. 


With this absurd combination of objects, the game is ready to be played. 
Visitors to the exhibition are invited to pick up a box and follow the instructions 


printed on the top (Fig. 6.4). The instructions send people on a journey travelling 
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Fig. 6.4 Boxes and 
Instructions Copyright © J. 
Townsley 2019 


through the space from coloured mat to coloured mat. There are 6 sets of instructions, 
all presenting a different order of coloured mats to visit: 


(1) go to the closest BLUE mat; 

(2) go to either a RED or to a BLACK mat; 

(3) go to another mat that is the same colour as your present one; 
(4) goto any YELLOW mat; 

(5) repeat forever from rule one above. 


Players physically ‘flow’ through the space of the gallery, moving according to 
the rules on their box. The brown or white boxes contain an everyday object: a knife, 
a fork or a spoon. When individuals meet at a mat, they open the box to compare the 
contents. Depending on the ‘local’ circumstance around the similarity or difference 
of those contents, players encounter one of two consequences: 


1. They continue to flow around the game: 


e this happens when their compared objects are different (non-repeated)—then 
the objects are exchanged and the flow continues according to the rules. 


2. They stop and exit the game: 


e if the objects are the same (repeated) then players are asked to leave their box 
and object behind, placed on a tower at the point of coordination. 
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Fig. 6.5 Players of the game Copyright © J. Townsley 2019, photograph Pierre-Stuart Rostain, 
included by permission 


In this way a ‘causal flow’ happens, when non-repeated coordination occurs, 
objects are exchanged and continue their journey, independent of player or box— 
flowing on, to continue the FOREVER-DO. Alternatively, when repeated coordina- 
tion occurs (people meeting with the same objects), the flow is halted and boxes build 
up around the mats. Over time the number of boxes in the towers gather randomly 
around the mats, infesting the space and forming a sculptural installation (Fig. 6.5). 

The installation makes visible a poetic data trail of coordinated human interac- 
tions, recording the incidence of two people meeting at a mat, to find that through 
random chance, their boxes contain identical objects. The coloured doormats act like 
nodes in a giant network. Receiving the flow of information as people and boxes 
move from mat to mat, enabling coordination and a place for human interaction 
(Fig. 6.6). 

As the box-towers grow, data may be extracted from the game in many different 
ways. The piles of white and brown boxes could be interpreted as binary code. 
The number of boxes clustered around a coloured doormat also give that mat an 
integer value. From this data, new digital, conceptual and physical constructs may 
be developed. Reorganised data is used to form a new sculpture: the FOREVER-DO 
Infestation. Alternatively, a new event or happening could be derived as presented 
in the FOREVER-DO Game 2. In other words, a new fishing expedition can begin 
(Figs. 6.7 and 6.8). 

More generally, the gathering, reorganisation and extraction of information, as 
presented in the FOREVER-DO Game, is indicative of the way data flows today. Data 
(collected in a myriad of ways) flows forever through our digital systems. Systems 
and data unimaginably vast in form, yet invisible in time and space. This data can be 
organised and extracted in whatever way society chooses: it can be reorganised and 
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Fig. 6.6 Players of the game and the box-towers Copyright © J. Townsley 2019 included by 
permission 


extracted to help us do great things for humanity, such as understand diseases; or it 
can be used less responsibly, to smartly target us to purchase more mass-produced 
goods. 


6.4 Petri Nets, Communication and the Bucket Chain 


The structure of the FOREVER-DO Game takes its principles from Petri Nets theory, 
in experiential and analogue terms it makes visible a form of communication that is 
happening across the interface of our physical and digital worlds. The installation 
presents a data trail made from the flow of people around a room, flowing until random 
instances of coordination happen between players. This has resonance with Petri Nets 
theory, in its desire to rationalise information flow as a resource in communication. 

The history of Petri Nets can be traced back to the Ph.D. thesis of Carl Adam Petri 
in 1962 [6]. His aim was to establish a new theory of communication-based on two 
assumptions: 


(a) there exists an upper limit on the speed of signals; 
(b) there exists an upper limit on the density with which information can be stored. 


These assumptions can be considered as natural ones today, but they were not so 
natural in the 1960s, when the theory of abstract computing machines (automata) was 
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Fig. 6.7 Image of towers 
around a mat Copyright © J. 
Townsley 2019 included by 
permission 


developing rapidly and, sometimes without regard for its potential limits in actual 
hardware realisations. 

As a consequence of these natural assumptions, Petri analysed the subject of 
‘information flow’ in a new way, based on the relations between the structure and 
behaviour of automata and on the role of information as a resource in communication. 
In his view, communication becomes an organised activity involving automata and 
people. Communication with automata, the title of the thesis, refers precisely to this 
organised activity where ‘with’ has the double meaning of ‘between automata and 
people’ and ‘using automata as communication medium’. Considering automata as 
organised communication mediums, as we do today through the Internet for example, 
Petri opened the way to new applications of the discipline ‘pragmatics’, as a branch 
of linguistics, in view of a modern theory of communication systems [7]. 

Petri’s bucket chain is a simple example of net theory and one way of rational- 
ising data flow. It presents a scenario of coordinated behaviour between firemen 
extinguishing a fire. Figure 6.9 shows figures coordinating a task to carry water from 
a tank to a fire, using a chain of buckets. 

The sequence in Fig. 6.10 explains how the coordination of behaviour (the figures) 
and the flow of data (the water from the tank to fire) operate. The theoretical net 
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Fig. 6.8 Image of towers around a mat Copyright © J. Townsley 2019 included by permission 


Fig. 6.9 The Bucket Chain (drawing from Ferigato’s blackboard, 2019) Copyright © J. Townsley 
2019 
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Fig. 6.10 The Bucket Chain and its unfolding in space and time (Ferigato 2018) Copyright © J. 
Townsley 2019 


‘unfolds’ through time and space to illustrate how the coordination of the figures 
putting out the fire operates. This is Petri’s clever graphical way of explaining simple 
net theory through the bucket chain scenario. 

As the information, in this case, the water in bucket B, flows smoothly from the 
water tank to fire, it is also possible to see how each fireman moves left and right 
until they meet to exchange the buckets (Fig. 6.11). 

The proposition: ‘A walks in the direction right’ can be represented, following the 
formalism of Petri’s Occurrence Nets, by a circle, representing in an abstract space 
the persistence of the proposition. 

Importantly, the truth (or value) of the proposition ‘A walks in the direction right’ 
is changed by the occurrence of events. For example, as in Fig. 6.10, two men 
walking in opposite directions along the same line will eventually meet at points in 
space represented as squares in Fig. 6.11. This ‘meeting event’ changes the respective 
directions of the two men, as if they were bouncing balls. So, we can represent a 
chain of firemen extinguishing a fire by carrying back and forth buckets of water in 
an abstract way, as in Fig. 6.11. 
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Fig. 6.11 The bucket chain WATE FIRE 
as a net (Ferigato 2018) 
Copyright © J. Townsley | ® 
2019 | 
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The net presents events, the meetings between two firemen, or, at the edge between 
the first fireman on the left and the reservoir of water and the last fireman on the right 
and the fire. Note that meetings between firemen do not happen in exactly the same 
place, nor are they regulated by any definite intervals of time. 

This is the principle illustrated in the game, people flowing from coloured mat to 
coloured mat according to a set of rules on a box. They are moving through the space 
as if in a deep fog, meeting other players outside of a determinate system of time and 
space. The rules written on the box propel the player according to a networked system, 
but when another player is encountered, the meeting event happens outside of the rule 
of a clock or defined by a specific determination of place. Also, the proposition for 
flow is changed with each encounter as objects are exchanged. In our digital world 
information also flows forever, it can be extracted at any time or place that interaction 
happens. This model also presents the basic principle of artificial intelligence, where 
the information has the ability to change according to an occurrence or meeting event. 
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6.5 Theoretical Context: Design and Analysis 
of (Coordinated) Communication Systems 


computer technology supplies us not [just] with a medium for artificial intelligence nor with 
a machine which may be used solely for computation, but with a medium for communication 
and for strictly organised information flow, a medium which may induce major changes in 
the modalities of co-operation between human beings [5]. 


The contemporary designer of communication systems is forced, in his profession, 
to consider relations between computers and society that are not yet consolidated 
in a unique practice or competence. Along the history of design of communication 
systems, plain consideration of measurable components like memory, throughput 
and availability, has shown its weakness in respect to less measurable subjects like 
delegation of tasks to computers, keeping control on copies of data and identify data 
as belonging to a given category. 

Petri proposed the first set of twelve communication disciplines that are today 
still a valid starting point for designing and analysing communication systems. 
He addressed the design of communication systems from a wide perspective, as 
coordinated activities involving humans and computers. 

The ‘Communication Disciplines’ were presented in September 1976 during a 
lecture given at Newcastle University and later published in ‘Computer Systems 
Design: Proceedings of the Joint IBM-University of Newcastle upon Tyne Seminar 
(B. Shaw editor). He stated: 


I tried to classify practical problems from my long list, according to my stated viewpoints 
of the role the computer [5]. 


His categories were: 


Synchronisation 
Identification 
Copying 
Addressing 
Naming 
Cancellation 
Composition 
Modelling 
Authorization 
10. Valuation 

11. Delegation 

12. Reorganisation 


SSO a a E 


His principal aim was to look into the future of computer systems and their 
applications, in order to support humanity to develop towards a brighter future. His 
conclusion was that computer system design should ideally be developed to ‘organise 
information flow’ in order to support ‘cooperation between human beings’. As the 
true maverick he was, Petri proposed a list that he hoped computer systems developers 
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could potentially use to ‘concentrate our attention on’ building a computer system 
that would enhance human cooperation and solve some of the problems he foresaw 
for a computerised world in the future. 


6.6 Fishing the Net of the Communication Disciplines 


To identify some of the processes and functions operating within the FOREVER-DO 
Game, we will use a selection of the 12 Communication Disciplines (highlighted in 
bold) as critical tools. In doing so we are also contextualising the game within a 
broader critical framework. 

Petri states that the discipline of identification ‘covers the question of pattern 
recognition’ [5]. The installation is a result of pattern recognition, specifically, a 
record of all the players whose synchronised meetings resulted in the identification 
of copied objects. Identifying sets of objects also indicates a destination at a mat. This 
is most obvious as the composition of stacked boxes grows around the mats in the 
room. Petri proposes that composition ‘is concerned with determining the structure 
of documents relative to a material or conceptual carrier’ [5]. In traditional art 
terms, there is a strong correlation here between ‘function and form’ in material 
composition. The towers are subject to the rules of the game, the rules act as a 
‘conceptual carrier’ determining the number of boxes in a pile. This takes the form 
of an ever-emerging flow of data, offered through time, as boxes move around the 
installation. 

Contemporary art escapes the present not by resisting the flow of time but by collaborating 

with it. It produces artistic events, performances, temporary exhibitions that demonstrate the 

transitory character of the present order of things and the rules that govern contemporary 


social behaviour. Imitation of the anticipated future, may manifest itself only as an event not 
as a thing [4]. 


Groys’s observation is a good way of thinking about the game, especially in the 
many ways flow can be identified. Flow happens in the reorganisation that ensures 
the slow evolution of the installation. Petri talks about reorganisation in relation to 
mechanised tasks within a nuclear power plant, he highlights the potentially ‘dis- 
astrous’ consequences if the flow of tasks and information are not adequately reor- 
ganised to meet changing situations. In the game, reorganisation is the process of 
becoming, an ever-changing contingent object. Reorganisation is embedded within 
the flow of the game and also enforced by external contingencies. 

The idea of artwork adapting to contingent conditions is not new, artwork can exist 
for just a moment within a definitive contingent space, temporary artwork such as: 
Allan Kaprow’s happenings and environments, or the installations of Doris Salcedo, 
Olafur Eliasson and teamLab et al. Martha Buskirk considers “The Contingent Object 
of Contemporary Art’ in her book by the same name [8]. She refers to the artwork 
Gnaw (1992) by Janine Antoni where the artist chews, over many days, two 600- 
pound cube’s, one made of chocolate, the other of lard. The action of gnawing results 
in a constant reorganisation. Buskirk says: 
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Thus the experience of the work, including the relationship of its components to one another, 
changes depending on when and in what part of the cycle one sees it, and also diverges from 
photographic records of its appearance [8]. 


This is pertinent to the FOREVER-DO, not only from the position of a passive 
viewer, but also for the player who is not only witnessing the reorganisation of boxes 
in a room but accepting active agency within that reorganisation. This implicates 
the player as an author within the ongoing realisation of the game. Petri says of 
authorization: 


this discipline is concerned not only assigning and schematically representing access rights 
but also with scheduling obligations [5]. 


By thinking about authorization in this way, a parallel can be drawn between 
authorization within a system, such as Petri indicates, and authorship commonly 
placed within art production, the ‘authorial’ process of an artwork. If, within the 
FOREVER-DO Game, authorship is shared across participants, objects, systems 
rules and networks, it cannot be solely located. Rather, obligations are shared, there 
is no single owner or receiver. Petri died in 2010 yet his inspiration is still implicit in 
the game, an authorial obligation through time. Authorial agency is delegated across 
all the agents of the game, flowing across the human, object and conceptual realm. 

In this way Delegation is forming an ever-changing collaboration between the 
mechanics of the game and the society of people who participate in it, Petri identifies 
in his networks a ‘delegation of tasks from one agency to another’ [5]. 

This shared social context is fluid, outside of a singular vision. There is, however, 
a fixed position in the naming of the groups of objects: a knife; a fork; a spoon. 
‘Naming is understood as denoting objects structures’ [5, p. 177]. Naming—becomes 
the subject of coordination, the similarity or difference, triggering the selection or 
transfer and the cancellation of the player in the game. As players track objects in 
the game they also have an experience of a route travelled and can anticipate a future 
route to travel. The pattern of flow through the network is in this way an experiential 
one, as infinite possibilities for coordinated behaviour are generated through random 
instances. 

The synchronisation is ‘partially’ controlled by the system mechanism, which 
embraces chance, ensuring that synchronicity happens outside of a deterministic time 
or place. In this way, the game is mirroring Petri’s premise that synchronisation is 
based on the ‘partial ordering in terms of causality as opposed to an ordering to 
time’ [5]. This mirrors Groys’s observation on flow being collaborative of time rather 
than fixing it [4]. 

Value, such as it is in FOREVER-DO, resides in the moment of participation 
or observation, not in a fixed object. Petri’s consideration of valuation states: ‘the 
instant the information is registered by the observer it already loses its value because 
it may not be presented anew.’ [5]. With this definition, valuation offers a very tricky 
problem to the traditions of an art market. Traditionally value in art has been placed on 
the object but, in the game, creative value is in the performative relational aesthetic of 
the work; it is essentially objectless, a process rather than an object. In this way, value 
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is transient, unable to be ‘presented anew’ a data value, that can only be experienced 
within the moment. 

In an attempt to contextualise art made in today’s global context, the French 
curator Nicholas Bourriaud coined an art movement, the Altermodern. He defined it 
as a moment when it is possible to produce something: 


from a vision of human history as constituted of multiple temporalities ... a positive vision 
of chaos and complexity [1]. 


In the Tate triennial exhibition by the same name, ‘Altermodern’ (2009). The 
triennial presented artworks from artists such as Marcus Coats, Mathew Darbysher 
and Franz Ackerman whose artworks offered a document, narrative or engagement 
with change. Earlier in his book ‘Relational Aesthetics’ (1998), he offers examples 
of art practice as a game: 


Artistic activity is a game, whose forms, patterns and functions develop and evolve according 
to periods and social contexts; it is not an immutable essence [1]. 


Art practice, based on the relational form, is presented in FOREVER-DO as a 
meeting point of exchange, a physical human interaction, within a networked system. 
In our digital age, these exchanges are often an opportunity for data mining. Points, 
relational in time and space, where data is gathered, producing much of our big data; 
data that is held outside of our physical realm. 


The role of artwork is no longer to form imaginary and utopian realities, but to actually be 
ways of living and models of action within the existing real, whatever the scale is chosen by 
the artist [1]. 


The FOREVER-DO Game also represents a ‘model of action within the existing 
real’. Big data, whose functions control us beyond our everyday perception, is here 
and is real. And yet, the otherness of the nature of big data, its invisible procedures, 
make it hard for us to perceive our own individual relationship with it—our essential 
role within the phenomena of big data systems. Without us, our relational interac- 
tions, big data would not exist while at the same time human life is becoming more 
dependent on the information big data feeds back to us. In reality, perhaps, we are 
already cyborgs in a close and symbiotic relationship with big data itself. 

The art historian Edward A. Shanken, in his essay Art in the Information Age 
(2001), considers work by artists such as Hans Haacke and Joseph Kosuth, he says: 


‘meaning and value are not embedded in objects, institutions, or individuals so much as they 
are abstracted in the production, manipulation and distribution of signs and information’ [9]. 


The process of the FOREVER-DO offers a distribution of signs and informa- 
tion, through the search for repetition. Repetition here is the key to unlocking the 
FOREVER-DO Game. It is the copying of objects (two knives, forks or spoons) that 
signifies the end of the player’s participation, while the process of the game contin- 
ues. Copying for Petri is a ‘message—occurrence in a definite pragmatic context’ [5]. 
In the game, this is a process held within an unfolding never-ending chain of events, 
Nietzsche like in its eternal return. In ‘Thus Spake Zarathustra’ (1917) Nietzsche 
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describes two eternal paths ‘This long lane backwards: it continueth for eternity. 
And that long lane forward—that is another eternity’ [10]. Most importantly the 
paths come together at a gateway: ‘The name of the gateway is inscribed above: This 
Moment’ [10]. 

In the game, the most significant ‘moments’ are the coordinated meetings accu- 
mulating to model the towers of boxes. Petri utilises the discipline of modelling 
to guard against rigidity of implementation of established systems across different 
fields. He states: 


our ‘models of thinking’, tend to gain some illegitimate independence once they have proved 
successful on a particular field, and are then — per analogy but without care — transferred to 
other fields [5]. 


Perhaps this is a warning against the transformative analysis between Petri’s dis- 
ciplines (as pertinent to computer systems) and the FOERVER-DO Game. However, 
the very temporality of the FOREVER-DO Game can perhaps embody modelling, 
where: 


we are able to deal with mathematical models in which notions of a temporal ordering is 
replaced by that of an ordering in terms of causality [5]. 


Whether you are an active participant in the game or a viewer on the periphery 
of the game, you are in communication with the model of the game and its objects. 
This is not a fixed exchange but a communication flow (a causality), with all agents 
causally influencing one another in a relational sense. 

Nicolas Bourriard draws on such symbiotic forms when he proposes that through 
the act of ‘inclusion’—the art object is causally significant to the participation of the 
viewer. Indicating that there is an energy exchange between the art object and the 
participant. 

The game is a destination in flux, a place directed by the rules, e.g. “go to the closest 
blue mat’. Petri offers a fluidity of destination with his discipline of addressing: ‘by 
this we mean the description of routes or systems of paths through the net of channels 
and agencies’ [5]. This flow of address as a destination, is an agency of the ever 
becoming. Addressing within the flow of the game drives it on to new beginnings, 
this chimes with Groys when he says: 


Being immersed in the flow of things, one cannot return to previous moments in time or 
experience the events of the past [4]. 


Addressing also happens in a subtler way too, as social address between players 
in the game. The form of address has a systemic origin beyond the game, residing 
in the social etiquette of human behaviour. This presents us with a wider network of 
behaviours and systems. Petri has proposed that his net systems could have a role 
in conflict resolution. He suggests, nets could help reconcile complex human prob- 
lems, emotionally driven problems, beyond the mechanical, systematic or digital, by 
offering clear alternative channels of communication. 
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6.7 Conclusion 


Art does not predict the future, but rather demonstrate the transitory character of the present 
— and thus open the way for the new [4]. 


By thinking about art as a process or event, as relational, societal and social, a new 
place is offered for authorship and realisation. This is not only the post-modern 
reform of authorship, deferred or shared, but authorship within the moment of a 
contingent ever flowing process—the FOREVER-DO. 

This conceptual space made real through event also requires a temporal place. For 
the FOREVER-DO Game the place of the Palazzo Giureconsulti at Duomo in central 
Milan, and temporarily occupied by Milan digital week, is important. Milan Digital 
Week is itself a social construct, a temporal event into which interested parties’ 
(visitors and participants) occupy space temporarily. Visitors are in this way already 
complicit within the flow of events. 

The historically embedded definitions of art and art practice, particularly the 
traditional consideration of value in relation to object, are being challenged through 
utilising digital systems as integral tools. By visualising, in analogue terms, this 
relationship between the participant (the player of the game) and the system (the rules 
and network of the game), the FOREVER-DO highlights our individual complicity 
within the greater whole. Places of meeting within the game indicated by simple 
coloured doormats are also nodes within a networked system. When individuals 
meet through a flow of random incidents, objects are either exchanged or offloaded 
onto the data pile. The piles of boxes record the incidents of two humans meeting and 
randomly possessing a repeated object. This process presents a digital and analogue 
description of events through time and offers a visualisation of a similar scientific 
modelling that the Petri Nets identify. 

Petri’s Communication Disciplines, written in the 1970s still ‘concentrate our 
attention on’ building a computer system to enhance human cooperation and over- 
come some of the many problems brought about by our human digital interface. 
Problems for culture arising from irresponsible systems, technicians, trolls and cor- 
porations. We are all complicit in society’s broader data communications. As Petri 
outlined, we must all consider the relations between computers and society that are 
not yet consolidated in a unique practice or competence. The FOREVER-DO opens 
a discussion about the interactive trail we leave behind and the systems acting upon 
us. By reclaiming our digital identity, being aware of the symbiotic relation of self to 
our larger networks, maybe we can hack our future in a positive way, for the benefit 
of all. 


Acknowledgements The FOREVER-DO project and its resulting artwork (including the 
FOREVER-DO Game) is a collaboration between Jill Townsley (University of Huddersfield) and 
Carlo Ferigato (Joint Research Centre of the European Commission—JRC-Ispra, Italy). The project 
is supported by advice from the MC3 research group at the Milano-Bicocca University and Otolab, 
Milano. The project is funded by the EU Commission through its SciArt initiative. 


102 J. Townsley and C. Ferigato 


References 


1. Bourriaud, N.: Relational Aesthetics. Les Presses Du Reel (1998) 

2. Joint Research Centre (JRC). Available: https://resonances.jrc.ec.europa.eu/. Accessed Oct 
2019 

3. Joint Research Centre (JRC). Available: https://resonances.jrc.ec.europa.eu/datami. Accessed 
Oct 2019 

4. Groys, B.: In the Flow, Verso Books. London (2018) 

5. Petri, C.A: Communication disciplines. In: Shaw, B. (ed.) Computing System Design: Pro- 
ceedings of the Joint IBM—University of Newcastle upon Tyne Seminar, September 1976, 
pp. 171-183. University of Newcastle upon Tyne (1977) 

6. Petri, C.A.: Kommunikation mit Automaten, Dissertation, Schriften des IIM 2, Rheinisch- 
Westfälisches Institut fiir Instrumentelle Mathematik an der Universität Bonn. (Eng. transl. 
Communication with Automata, Technical report RADC-TR-65-377, Volume I, Final Report, 
Supplement I) (1962) 

7. Bernardinello, L., Cardone, F., Pomello, L.: L’eredit’a di Carl Adam Petri: dagli automi alla 
comunicazione, Mondo Digitale, febbraio, p. 175 (2016) 

8. Buskirk, M.: The Contingent Object of Contemporary Art. The MIT Press, Cambridge, MA, 
USA (2003) 

9. Shanken, E.A.: Art in the Information Age in LEONARDO, vol. 35, No. 4, pp. 433—438 (2002) 

10. Nietzsche, F.W.: Thus Spoke Zarathustra. Modern Library, New York (1917) 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Chapter 7 A) 
Searching for New Aesthetics: Unfolding ere 
the Artistic Potential of Images Made 
by the Scanning Electron Microscopy 


Anastasia Tyurina 


Abstract The visual arts have become a powerful tool for alternative approaches to 
scientific outputs, but it is crucial that both science and art cultures are aware of their 
interdisciplinary capabilities and limitations. It is necessary to differentiate between 
images captured by devices designed as resources for scientific investigation and 
images that exploit the ‘visual elements’ of scientific images. A great example of 
a device designed for scientific exploration is the Scanning Electron Microscope 
(SEM), which was introduced to scientific research in the mid-1960s. This chapter 
outlines the procedures, processes, and methodologies used in developing a body 
of studio work that investigates the artistic potential of scientific images made by 
the Scanning Electron Microscope (SEM) with a view to its possible social and 
cultural impact of this practice. It also outlines the developed theoretical findings and 
proposes that certain images made by the SEM can have esthetic value apart from 
that of scientific documentation. The use of artistic manipulations in experiments 
with the SEM fuses science and technology with art, and the SEM-based images that 
result provide a new meaning for scientific photomicrographs. 


Keywords Science-art - Visual arts - Photomicrography - Interdisciplinary + 
Microworld - Interactivity - Aesthetics - Self-organizing processes - Coffee ring 
effect + Alternate visualizations - Visual coalescing - Creative coding - New 
media + Scanning electron microscope + Water 


7.1 Artist Statement 


My art practice involves interplay between photography and scientific imaging, pho- 
tomicrography in particular. Originally being a technical discovery, photography has 
been widely used in almost all fields of human activity, acting as a research tool and 
as an independent artistic direction. Modern times offer “modern” interpretations of 
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scientific photographs and attitudes toward them. Scientific photomicrography is a 
powerful tool for receiving and storing information and for providing solutions to 
a variety of tasks in many areas of science and technology. At the same time, the 
artistic application of photomicrography is capable of revealing a set of complex and 
interrelated principles that underlay the materiality of the human environment. Pho- 
tomicrographs also expand human visual vocabulary, revealing principles of beauty 
which are difficult to access with the naked eye. 

Water is the main subject of my research project. I believe that an interdisciplinary 
approach is the most appropriate one for deepening knowledge about unique prop- 
erties of water and building a sustainable practice of water management through 
enhanced visualization of water contamination. My focus is on interconnected artis- 
tic thinking and by developing alternate forms of visualization, I aim to transcend 
disciplines and contribute to the new ways of seeing water. 

My visual artworks consist of series of images made by the SEM depicting the 
nature of water (water chemistry), which is invisible to the naked eye, through using 
the SEM utilizing the phenomenon of drop evaporation. However, it was not the 
purpose of this research to claim that the created images of water after evaporation 
are scientifically valid forms of documentation; rather, this research takes an esthetic 
approach to scientific photomicrography. 


7.2 Introduction 


This research was undertaken during my Ph.D. study at Queensland College of Art, 
Griffith University and involved esthetic approaches to scientific photomicrography. 
Scientific photography is commonly perceived as a way of recording scientific data 
through techniques such as photomicrography, high-speed photography, time-lapse 
photography, X-ray photography, and aerial photography, among others. 

Images made by the SEM are not photographs in the traditional sense; they go 
beyond what can be captured with light because the process of producing a picture 
is camera-less. Imagery produced by the SEM can confuse the viewer because the 
microscopic sample seems as if it is observed in the eye aperture when illuminated, 
and light seems to come from a particular illuminant. SEM photomicrographs are 
constructed out of pixels synchronized with a distribution map of the intensity of the 
signal being emitted from the scanned area of the specimen [1]. 

Interestingly, the last decades’ use of SEMs in creating scientific images formed 
a new, well-established visual culture within a variety of scientific disciplines. As 
Klaus Hentschel explains, it became “an image centered science in the sense of being 
even totally dependent on photographic images as basis of all further processes of 
inference” [2, p. 315]. 

By exploring the interplay between the indexical and iconic modalities in scientific 
photomicrographs, I try to imbue them with new meanings. Both art and science 
are experimental in nature. There are different ways of artistically representing and 
perceiving an object, some of which may be valuable for science. My practice aims 
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to draw attention to the qualities of water through enhanced visual details that aid in 
the interpretation of and differentiation between water samples. 

This project investigates how to reinterpret photomicrographic images made by 
the SEM of micro-scale drops of water after evaporation and thus turns scientific 
photography into an art form. 

In recent decades, there have been increasing concerns over the ecological man- 
agement of water. Waterway pollution is recognized as placing urban ecosystems 
around the world at risk. Rainfall that washes oils, metals, and nutrients directly 
from streets into rivers and seas is hard to treat [3]. This is a challenging problem for 
science and technology. The health and well-being of present and future generations 
are dependent on how quickly and well it is managed. 

Photomicrography has a particular potential to respond to this issue from both a 
scientific and a cultural perspective. SEM-made photographs are capable of visually 
representing features related to water composition and, in some cases, the contami- 
nation of water. At the same time, they can transmigrate from science into the domain 
of art and draw attention to water issues. 


7.3 Water Represented by the Scanning Electron 
Microscope 


The variety and frequency of the unusual properties of water are determined by the 
physical nature of its atoms and their association in the molecule and the group- 
formed molecules [4, 5]. The composition of water, even that which is entirely 
free from mineral and organic impurities, is complex and diverse because water is 
constantly in contact with many substances. 

During my experimentation with the water droplets collected from different 
aquifers, I aligned the scientific method of revealing water composition (the so-called 
‘Coffee Drop Effect’) with my artistic practice. The drop evaporation phenomenon, 
the so-called ‘Coffee Ring Effect’, has been the subject of studies in the last few 
decades. It was first mathematically described by Robert D. Deegan in 1997 as a 
natural model for studying the dynamics of self-organizing processes and is actively 
used in physical experiments [6]. When the liquid is ‘pinned’ to its contact surface, 
the liquid evaporating at the exterior edge is replenished by the constant flow of 
liquid from the interior to the periphery. This flow carries nearly all dissolved solids 
toward the edge [7]. 

During experiments for my project, the structure of the water impurities visually 
transformed, leading to a unique connection between the processes of evaporation 
and solidification. These two processes are shown in Fig. 7.1. This natural process 
of drying reveals the unique informative capacity of droplets through the shapes, 
patterns, details, and characteristics resulting from each water sample. 

At the University of Chicago Materials Research Center, scientists are explor- 
ing the driving mechanism responsible for this phenomenon that is found in many 
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EVAPORATION 


SOLIDIFICATION 


Fig. 7.1 Anastasia Tyurina, Drop Evaporation and Solidification Scheme, 2015. Image copyright 
© A. Tyurina 2019 


varieties of liquids. This research validated my process of exploring variations in 
water composition for my project. On its webpage, the Center states: “We have 
determined that ring formation is a ubiquitous and robust phenomenon. It doesn’t 
depend on the solute, the solvent, or the substrate so long as the solvent is partially 
wetting and volatile, and the contact line is pinned” [8]. 

An interesting observation made by the researchers is that “By controlling the 
speed and spatial variation of the evaporation, this model predicts that we can control 
the shape and thickness of the deposit” [7]. This was particularly important for my 
practice because it meant that experimenting with various modes of evaporation 
could lead to different visual formations of the drops after they dry. 

Specifically, for this research ‘science’ can be described as a process through 
which new knowledge about the world is built. The scientific process relies on the 
testing of ideas through experimentation. Scientific photography, as a tool, aims to 
support the scientific process by capturing data about the world for examination. 

Reflecting on the scientific findings, the primary purpose of my visual art project 
was to depict the inherent features of water that are invisible to the eye through using 
the SEM. To do so, I used the process of evaporation as an alternative and unusual 
artistic method of visually presenting the composition of water. My approach is 
unique in the specific way in which I use water to create images using the SEM. 
During experiments for my project, it became evident that the structure of water 
impurities is visually transformed after evaporation and reveals a unique connection 
between evaporation and solidification. This process of revealing the nature of water 
(water chemistry) allowed me to play with the process like an artist [9]. 

One of the technical requirements when taking a microscopic image with the 
SEM is that any object placed in the chamber of the SEM must be completely dry, 
because the SEM operates in a vacuum. As explained in Under the Microscope: A 
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Hidden World Revealed (1987), “Turning to biological specimens for the SEM, it 
used to be necessary first to fix and dehydrate them, and then dry them either on air 
or by the use of liquid carbon dioxide (‘critical point drying’)” (10, p. 199]. After 
the evaporation process, water is no longer a liquid; dry solids or other substances 
become watermarks that represent its previous composition. 

The main difference between ‘Artistic’ and ‘Scientific’ use of the SEM in this 
project is that the artistic approach has a focus on making aesthetic images. Within 
this is the substance of the scientific idea about showing water state and how it is 
changing. In my work the results of imagery processes become art pieces with both 
artistic and scientific applications (Fig. 7.2). 

My artistic intervention into a scientific process through experimenting with the 
SEM was a way to find out what different things my images could reveal about 
water to a viewer. There was an opportunity to explore some concerns related to the 
environmental impact of water pollution in an artistic context and to communicate 
the significance of water. 

By exploring the integration of art and science in this way, my project focuses 
on the use of scientific tools to create science-based art. However, it must be recog- 
nized that my approach was primarily one of artistic research and experimentation: a 
search for new aesthetics, exploring spatial and temporal dimensions, engaging with 
materiality, and involvement of modern technologies in the formation of esthetic 
experiences for the viewer. In my project, images are highly variable in what they 
show and how they show it because of changes in water content. 


7.4 Photomicrography as an Art Form 


Contemporary practices in photomicrography are making significant contributions 
to the dialog around aesthetics and artistic components in science. As far back as 
1963, in his article “The Esthetic and Pictorial Applications of Photomicrography” 
for The Photographic Journal, Douglas Lawson wrote: 


When mentioned within the hearing of some pictorialists they immediately think it is only for 
the scientifically minded, or the record worker. Nevertheless, the so-called record photograph 
can be made to look quite attractive. Some of you may remember Mr. H. A. Murch, one of 
our great pictorial photographers, once saying, ‘I do understand the desire to apply pictorial 
ideas in record work, which is a very different objective, and we ought to welcome such an 
application when it can be done without losing anything of the essential factual value of the 
record’ Photomicrography offers expression not in what the painter has already done but in 
what the painter cannot do. Because this application of photography is highly scientific there 
is perhaps a tendency to think of it as being without scope for the artistic application. May I 
suggest that a work of art in any medium can be the deliberate creation of unity, and nature 
through the microscope is one medium provides us with plenty of scope for such unity [11]. 


To effectively engage with photomicrography as a social phenomenon, it is crucial 
for an artist to demonstrate an understanding of its ‘scientific’ protocols of represent- 
ing. But will the viewer be familiar with what he or she sees in scientific images? If 
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Fig. 7.2 Anastasia Tyurina, Watermarks Series, 2015 (From left to right: Brown Lake, Brisbane 
River, Raby Bay, Rainwater, Mount Gravatt Pond, Mimosa Creek, South Bank Pond, Cylinder 
Beach). Image copyright © A. Tyurina 2019 


not, how will they perceive such images? Do the images need to be explained or are 
they capable of conveying messages in a different context? 

Artistic photomicrographs of water after evaporation call upon the viewer’s cre- 
ative ability to perceive previously unseen water composition after evaporation as 
well as to observe natural phenomena over and beyond the directly visible. 

My images transform the microworld to a macro-level and evoke an interest in 
water chemistry that is shown as being beautiful. This causes a dilemma for viewers, 
particularly because the gallery space is different from a laboratory. The captions 
for my photographs refer to the sites where the water samples were collected. It 
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is intriguing that they can resemble aerial photographs of topographical features of 
particular water reservoirs (Figs. 7.3 and 7.4). 

Today, the practice of visually pleasing photomicrography is more mainstream. 
The most popular international photomicrography competition is Nikon’s Small 


Fig. 7.3 Anastasia Tyurina, Photo documentation of the artistic installation for H2O + exhibition, 
PoP Gallery, Queensland College of Art, Griffith University, 30 March-16 April 2017. Image 
copyright © A. Tyurina 2019 


Fig.7.4 Anastasia Tyurina, Photo documentation of the Shifting the Posts exhibition, Webb Gallery, 
Queensland College of Art, Griffith University, 27 November—7 December December 2018. Image 
copyright © A. Tyurina 2019 
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World Photomicrography Competition, which dates back to 1975. Participating 
images are judged for “their scientific and artistic merits” [12] and represent a broad 
range of visual studies of the microworld. Some of these photomicrographs have alle- 
gorical titles, a fact that highlights the relation of artistic intentionality in producing 
images and the scientific objectivity underlying the concept of the competition. 

Participation in the competition is restricted to images made by optical micro- 
scopes of any kind, which are capable of revealing natural colors of objects. As 
mentioned above, the SEM apparatus is camera-less; there is no light involved in the 
process; it is not an optical instrument. Thus, colors cannot be reproduced. Rather, 
artificial colors can only be added with the help of graphic software such as Adobe 
Photoshop. While there is a whole range of possible manipulations available in Pho- 
toshop, it is worth mentioning the coloring tools. Through different algorithms, it is 
possible to apply different colors to different details of the image. There are many 
sources both in the technical literature and on the Internet providing scientists and 
artist with various tutorials on image coloring. But the author of the image makes the 
final decision of which color to apply to a particular point of the presented object and 
thus color choice is very individual. Experimenting with such coloring, I have tested 
two options: the coloring algorithm available in Photoshop and the software offered 
by Recolored, which was designed for the purpose of recoloring black-and-white 
photographs (Fig. 7.5). 


Fig. 7.5 Anastasia Tyurina, Colour 32, 2017. Image copyright © A. Tyurina 2019 
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7.5 Photographs or not Photographs? 


It is important to examine the process of image formation in relation to the imagery 
obtained by the SEM in modern microscopy. It is also crucial to distinguish image 
generation by SEM considering that the process is not one that is based in code. The 
image generation process can be seen as a work of high-energy electrons that create a 
range of different signals when hit the specimen. Converted into pixels signals result 
in image formation appeared on the screen. 

Technically, the SEM produces electron micrographs because the image is devel- 
oped by either secondary electrons (electrons ejected from the material by the incom- 
ing electron beam) or backscattered electrons (electrons from the electron beam, 
which have ‘bounced’ off the material into the detector). Dee Breger explains: 


Scanning electron microscopes don’t merely use electron beams to illuminate objects so 
small they can’t be seen by light. Since they are electronic devices, SEMs can manipulate 
isolated groups of electrons from the sample-beam interaction to create separate pictures 
(such as secondary and backscatter images) that contain different kinds of information about 
a single object. This variety can lead to a ‘compound reality’ or, since no version tells the 
whole story, a kind of ‘ironic nonreality’ [13, p. 11]. 


Images produced by the SEM are beyond light; captured by a focused beam of 
electrons, they are not photographs. The apparatus tries to recreate a reality that 
is not a visual phenomenon, which scientists then try to analyze through its visual 
representation: the photomicrograph. 

However, the most commonly used SEM electron detector, named Everhart- 
Thornley (E-T),! typically uses a material that produces light when an electron 
collides with it—a scintillator. 

Therefore, the SEM does use photons, but they are converted back into elec- 
trons, which are accelerated onto the electrodes of the photomultiplier, producing 
an increasing stream of electrons until the final collector is reached [14]. The image 
formation in scanning electron microscopy consists of the scanning system, the sig- 
nal detectors, the amplifiers, and the display [1]. A physical, material connection of 
the SEM apparatus with the studied object is gained by the interaction of these four 
elements (Fig. 7.6). 


'“The E-T detector operates in the following manner: when an energetic electron (10 keV energy) 
strikes the scintillator material (S), light is emitted. Light is conducted by total internal reflection 
in a light guide (LG) (a solid plastic plastic or glass rod) to the photocatode of a photomultiplier 
(PM). At the photocatode, the photons are converted back into electrons, which are accelerated onto 
the successive electrodes of the photomultiplier, producing an ever-increasing cascade of electrons 
until the final collector is reached” [14]. 


“A scintillator is a material that accepts incident high-energy electromagnetic or charged particle 
radiation and in turn uses that energy to fluoresce photons whose peak emission wavelength is longer 
than the wavelength of the incoming radiation. In the case of a SEM, the scintillator disc collects the 
secondary electrons that are produced as the electron beam scans the surface of the sample. These 
electrons are converted into photons which travel through the light pipe to the photomulitiplier tube 
(PMT) so that the signal may be amplified to the level required for viewing” [15]. 
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Fig. 7.6 Anastasia Tyurina, Scanning Electron Microscope Imaging Facility, 2016. Image 
copyright © A. Tyurina 2019 


During the imaging process, the energy exchange between the electron beam 
and the sample results in the reflection of high-energy electrons, the emission of 
secondary electrons and the emission of electromagnetic radiation, each of which 
can be measured by specialized detectors. The detectors interpret these signals by the 
algorithm and the resultant image appears constructed out of pixels. This algorithm 
is defined by each quantum of signal information but it is also implemented in the 
physical world in apparatus, computer, and screen [1]. In some sense, the SEM 
apparatus interprets (transforms) signals and visualizes signals. 


7.6 Scientific Images and New Media 


The artworks employ two kinds of medium, comprising two different applications of 
SEM photomicrography: still photomicrographs and interactive digital installations. 
They can be perceived differently in terms of the esthetic response they generate in 
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the viewer, but both draw attention to the qualities of water through enhanced visual 
details that aid in the interpretation of water samples. I use static SEM-generated 
photomicrographs of water, then apply a digital code which allows the image to be 
altered by the audience’s interaction with its touch-sensitive interface. Thus, anyone 
who touches the screen can create, display, and experience a ripple effect, which is 
very similar to the effect we can see and observe when we interact with water surface 
by touching or disturbing it. Interacting with the scientific photomicrographs in this 
way offers a layered meaning and can enhance the audience’s perception of scientific 
data, scientific photography, and water. 

Exploring the idea of ‘interactivity’ for my project, I looked for a method that 
allows viewers to interact with photographs by altering the work’s visual content. 
Utilizing the programming language Processing, I developed an algorithm which 
allows viewers to physically interact with my photomicrographs so that they become 
direct objects of manipulation. Using the random human touch of some areas of the 
developed algorithm alters the static image, which in some senses visually represents 
non-visual elements: scientific data. 

Through my research, I have explored different digital tools and realized that cre- 
ative coding suits my goals because this type of computer programming is designed 
to create something expressive instead of something functional. Using creative cod- 
ing tool such as Processing, it is possible to create art installations, projections, sound 
art installations, and much more within the context of the visual arts. 

The interplay between virtual ripple effects and the resemblance of physical inter- 
action with water can be seen as embodying the “artist—display—user” paradigm seen 
with regard to computer art [16, p. 10]. In their article “The Post-Display Condition of 
Contemporary Computer Art,” Toby Juliff and Travis Cox attempt to re-conceptualize 
the relationship between artistic intentionality, coding, interface, and user input. They 
suggest that the relationship between the artist and computer code should be recon- 
sidered so that the paradigm becomes “artist-code—display—user.” They state that 
“the code takes input from the user and subjects it to internal semiosis between dis- 
tinct elements of the code, before being output to the display. This code, having been 
written by the artist, or an agent of the artist, contains within its structure an inbuilt 
intentionality, a way of approaching input that is integral to conveying the intended 
meaning” [16, p. 10]. In line with artists using code to activate new experiences and 
with Juliff and Cox, who argue that the code is not only a product of intention and 
meaning but also a producer of them, I seek in my practice to explore the role of 
code in the production of meaning, in particular for the scientific photomicrograph 
of natural phenomena. 

In the series of live images for my project, the photomicrographs displayed on 
the screen seem to be still when viewed from a distance; yet, as the audience comes 
closer and start touching the screen, the ripples appear. 

The photomicrograph transforms into a live picture, producing varying forms of 
ripple that seem both ordered and random at the same time. Such transformations 
occur continuously until the user-viewer stops touching the screen (Figs. 7.7 and 
7.8). 
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Fig. 7.7 Anastasia Tyurina, Rainwater (detail) 2016, Digital mixed media. Image copyright © A. 
Tyurina 2019 


Fig. 7.8 Anastasia Tyurina, Rainwater (detail) 2016, Digital mixed media. Image copyright © A. 
Tyurina 2019 


Interacting with the image in this way transforms the work into something that 
transcends disciplines. It offers a layered meaning, providing audiences with the 
opportunity of experiencing the fluid and animated qualities of the effects that con- 
nects with these qualities in the subject matter: water. The addition of the interactivity 
and animation introduced by the ripple effect is the expanded mode of ‘reading’ or 
appreciation. As well as offering a visual engagement, the work offers an embodied 
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Fig. 7.9 Anastasia Tyurina, Colour 32 (detail) 2017, Digital mixed media. Image copyright © A. 
Tyurina 2019 


engagement: an important connection to the material significance of water in our 
lives. 

Colour 32, one of the artworks, portrays a drop of Brisbane River water. This 
artwork was exhibited at various venues displayed as both interactive work and 
video at screens of different scale (Figs. 7.9, 7.10 and 7.11). 

Such digital installations allow the transformative power of water to be explored; 
they offer new perceptual experiences and, by the artist-code—display—audience inter- 
action, are capable of providing a new meaning for scientific images through their 
ever-changing visualization. 


7.7 Conclusions 


Not every image is art, and art is far more than just an image. Originally a technical 
discovery, photography has been widely used in almost all fields of human activities, 
acting as a research tool and as a form of artistic practice. Scientific tools have 
brought new ways of seeing the world. Although usually reserved for scientific use, 
such technology is now being used in the creation of art. 

The primary purpose of this research was to show that the artistic use of SEM- 
made photomicrography can shift the visual outcomes of scientific photomicrography 
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Fig. 7.10 Anastasia Tyurina, Colour 32, North Spine Plaza, Media Art Nexus, Nanyang 
Technological University (NTU), Singapore, 2017. Image copyright © A. Tyurina 2019 


Fig. 7.11 Anastasia Tyurina, Colour 32, public screening, Adelaide Festival Centre, 2018. Image 
copyright © A. Tyurina 2019 


to function within the context of art. Drawing on literature, visual analysis, and 
theoretical findings, it was possible to develop my artistic practice using the SEM to 
visualize the microworld of water. As Breger states, the microworld exists beyond 
our cognition; its beauty is hidden but it can be explored using appropriate tools, 
such as the SEM [13]. 
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Not many people have access to complex research technologies that can widen 
their daily experience such as the SEM. Mostly, people passively receive scientists’ 
dry results and the interpretations they are exposed to. My work in the field of 
photomicrography aims to expand human visual vocabulary, revealing principles of 
beauty which are typically difficult to otherwise access. Thus, my artistic intention is 
to expand the visual representation of the composition of water, its chemical features, 
and their patterns after evaporation. My aims are to also communicate issues related 
to the ecological management of water and to raise and deepen responsible attitudes 
toward consuming and managing water resources. 

During this project, I have placed (both virtually and physically) scientific images 
into a ‘hostile’ artistic environment. In the gallery space, the significantly greater 
scale of photomicrographs, their impeccable details, the artistic materials used, and 
the digital, still and interactive modes of presentation uncover the novel artistic 
potential of the scientific image. This leads to the image being disconnected from 
pure data and objectivity and allowing the viewer to perceive it as art, helping them 
to interpret photomicrographs of water in different ways. Most importantly, through 
these images, I hope to uncover the horizon of meanings previously unseen. 

Today, many institutions and organizations host scientific photography and visu- 
alization competitions (and associated exhibitions), which encourage scientists from 
different disciplines to contribute. The images are not only judged by profession- 
als (scientists and photographers) but often the audience is also invited to evaluate 
the competing images. By doing so, all three groups apply their own esthetic judg- 
ment criteria. This illustrates how scientific images can be evaluated from different 
contexts. Examples of such events include the annual the Visualization Challenge, 
National Science Foundation; the Wellcome Image Awards, Wellcome Trust; Nikon 
Small World Photo Competition; The Royal Photographic Society International 
Images for Science, just to name a few. 

This popular trend indicates the growing interest in the visual coalescing of science 
and art through photography, particularly through photomicrography. Nevertheless, 
the idea that the photographic scientific image can be artistic and that the arts can 
serve science and vice versa still generates skepticism, and the debate will surely con- 
tinue. Regardless, it is obvious from the discussion that the arts have the potential 
to reveal important aspects of the complex world in their own way, which is com- 
plementary to what scientists do and may be useful for both disciplines. Scientific 
photography aims to record and illustrate data and experiments that differ according 
to specific disciplines. Although the main purpose of scientific photography is to 
convey accurate information rather than beauty, its ability to record material in addi- 
tion to that which is merely informative allows it also to serve expressive, subjective, 
and esthetic purposes. 

The new definition other than photographs (or photomicrographs) for SEM images 
should be sought considering the nature of the process of image generation by the 
SEM. This is particularly interesting due to the complex nature of the apparatus. 
Different ways in which images made by SEM can be interpreted create layered 
meaning in them especially if they are given in the artistic context. 
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Chapter 8 A) 
Interspecific Interactions: Interaction ciecie; 
Modes Between Sound and Movement 

in Collaborative Performance 


Manoli Moriaty 


Abstract Collaboration between composers and choreographers is an approach 
possessing a long history and expansive list of creative outcomes. In recent years, 
such collaborative endeavours have further engaged with knowledge and practices 
from scientific disciplines, consequently giving rise to an emergent field of research 
focusing on novel modes of interaction between the expressive media of sound 
and movement as facilitated by new technologies and methodologies; from ana- 
logue proximity devices permitting dancers rudimentary control over recordings in 
the 1965 Variations V, to Marco Donnarumma utilizing biophysical sensors and 
machine learning algorithms as means of maintaining detailed command of digi- 
tal signal processing (DSP) through specific body postures in the 2016 Corpus Nil. 
Whilst the contributions by this relatively new transdisciplinary field have produced 
practice and research outcomes of unquestionable value, its main focus concen- 
trates on the expression of sound through movement, with lesser emphasis placed on 
perspectives of practitioners utilizing interactive sound as means of informing the 
arrangement of movement. This chapter aims to reflect those perspectives through 
the research conducted during my collaborations with dance practitioners, where we 
have together examined areas such as each performer’s role while operating inter- 
active systems, employing different modes of system operation according to the 
desired determinacy of resulting material, and simplification of discipline-specific 
language when engaged in polydisciplinary collaboration. The contribution of this 
research concerns a novel taxonomy of interaction modes informed by the biological 
phenomenon of symbiosis. Defined as close and persistent relationships between 
organisms of different species, symbiosis manifests in three core types—mutual- 
ism, commensalism, and parasitism—with each type determined according to the 
change in fitness outcome experienced by each of the engaged organisms. In the 
context of my collaborative practice, the biological notion of fitness outcome is 
interpreted as the expressive range allocated to each performer and their respec- 
tive media, while simultaneously relating to the level of determinacy from the score 
and choreography. The symbiotic modes of interaction are firstly described through 
examining three contemporary precedents, each showcasing distinct approaches in 
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collaboration and use of gesture recognition technologies (GRT), followed by a 
practical demonstration of each mode by its activation during a practice outcome 
resulting from my collaborations with two dancers, including a detailed analysis of 
the developed interactive system and its principles of its operation. 


Keywords Polydisciplinary © Collaboration - GRT - Interaction - User interface - 
Symbiosis 


8.1 Introduction 


For the better part of the past decade, a significant portion of my practice and research 
has been concerned with polydisciplinary (see note 1) collaboration in performance 
practice, with a particular focus on the processes by which practitioners expressing 
through sound and movement engage in collaboration. Since its early conception, 
my approach in examining such engagements stems from a perspective of biological 
associations, namely the phenomenon of symbiosis. My preoccupation with sym- 
biosis is fuelled in equal parts by the phenomenon’s scientific significance towards 
evolution, and the way it has informed seminal philosophical observations on social 
organization among humans [1]. The outcome of this artistic research resulted in 
two publications [2, 3] with symbiosis informing a framework through which I went 
to analyze the process of and the social dynamics observed within the collabora- 
tive engagements conducted as part of my practice, largely concerning live perfor- 
mance works involving interaction between music and dance. At the same time, 
however, these publications provided scarce details on the technological means used 
to facilitate the interaction between our respective expressive media. Considering the 
fundamental role of technology in developing our joint practice outcomes, further 
reflection on the conceptual debate between symbiosis and performance practice 
complemented the existing framework for interaction between disciplines and prac- 
titioners resulted in findings that appear salient towards identifying distinct modes 
of interaction between the expressive media of sound and movement. 

The findings presented in this chapter have been reached through a Practice 
Research methodology, and are presented through a process of analytic autoethnog- 
raphy, that is the active reflection of one’s experiences in creative work through the 
wider context of associated practices. The contribution of these findings on technol- 
ogy focus on the application of well-established GRT devices and associated software 
suites within a novel set of interaction modes informed by the taxonomy of symbiotic 
relationships, with the activation of these modes presented through both precedent 
examples, as well as a representative outcome developed through my collaborative 
practice. In other words, rather than aiming towards creating new technologies for 
creative expression, this research aims to present new ways of appropriating existing 
technologies, particularly in manners accessible to practitioners of diverse levels of 
technological proficiency. 
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My motivation towards this research approach stems from my previous practice as 
a performer of Electronic Dance Music, where the limited possibilities presented by 
analogue disc jockey equipment were a driving force to develop new ways for sonic 
manipulation. This exploration continued during my first steps within sonic arts, 
albeit this time through predominantly digital tools, with the addition of modulation 
devices (e.g. Envelope Generators (EG), Low Frequency Oscillators (LFO), Step 
Sequencers), as means of predefining and accurately recreating certain sonic gestures 
in the performed material. However, the accuracy of digital devices made away with 
the organic and ephemeral feel borne of the imperfections of human movements. This 
observation coincides with my first forays into collaboration with dancers, as well 
as my initial experimentations with GRT in sonic performance through the widely 
used Nintendo Wii Remote (or Wiimote). These early experiments were conducted 
in collaboration with contemporary dancer Shona Roberts, an already close personal 
friend, with whom we began exploring ways for motion sensors to manipulate and 
modulate sound (see Fig. 8.1). 

At the time, my main focus was to explore the potential of using physical move- 
ment as means of affecting, guiding, and enriching sound, with the connection 
between the two expressive media being a lesser concern. In essence, my perspective 
at the time could be described as sonic-centric, where the performer’s actions were 
utilized as merely another source of modulation to augment the previously used 
devices. Contrary to reducing my collaborator’s contributions to the programmed 
actions of a machine, this approach allowed Roberts unrestricted range of expression 


Fig. 8.1 Performance with Shona Roberts at Anatomy, Edinburgh Summerhall, April 2014 
(Copyright Richard Dyson, reproduced by permission) 
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through her choreography, while I enjoyed similar liberty in creating sonically mean- 
ingful interpretations of the dancer’s arm movement through the data captured by 
the pair of Wiimotes. 


8.2 Interactivity in Performance—Origins and Recent 
Developments 


On reflection, this mutual independence during our earliest experiments with Roberts 
echoed the collaborative relationship between composer John Cage and choreogra- 
pher Merce Cunningham, who in developing their joint works would ‘intentionally 
segregate the creation of the sound from the creation of the movement until the per- 
formance’ [4]. Dubbing this relationship ‘autonomous complementarity’, Andrew 
Uroskie explains that this separate development of disciplinary material was an effort 
in alleviating the dependency of movement to embodying sound in a manner that 
was to an extent literal, and certainly perceivable by the audience: 


Music was understood to govern, implicitly or explicitly, the movement of the dancers. 
Choreography, within this propulsive conception, was a kind of musical interpretation, judged 
on its ability to form a singular synaesthetic coherence in the experience of the audience. [5] 


Challenging the traditional notions of audience experience in the work of Cage and 
Cunningham is further evident from the presentation format of the Black Mountain 
‘Happenings’, a series of events where the combination of music, dance, and visual 
arts was such as to allow each audience member to experience a unique perspective 
[5, 6]. The two artists’ focus was shifted from the experience of the audience to that 
of the performers, and the manner in which the latter embodied the received aural 
and ocular stimuli. 

Having explored independence between sound and movement throughout their 
work in the 1950s, in 1965 Cage and Cunningham developed Variations V, arguably 
the first performance to feature technologically facilitated interaction between music 
and dance. Working alongside the artists, engineers Billy Kliiver and Robert Moog 
developed two distinct approaches in motion-activated control of music equipment. 
Respectively employing photosensitive cells and proximity antennas [7], the system 
was designed as to react to dancers’ movements by operating the transport con- 
trols of several tape machines storing Cage’s sound palette. As such, the notion of 
sound governing the ‘propulsion’ of movement was all but eliminated in Variations 
V. However, as Uroskie points out, the dancers were not in conscious control of 
the sonic palette, which would imply that ‘one model of subordination would have 
merely been exchanged for another’; instead, movement was utilized to ‘set a cer- 
tain train of sonic events in motion’ [5]. In other words, while dancers performed 
Cunningham’s precisely rehearsed choreography [7], the sounds triggered from the 
interaction between movement and the two motion-detecting systems resulted in an 
indeterminate sonic outcome, an approach in music score that is often associated 
with the practice of Cage. 
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With the benefit of more than fifty years of technological developments, a mod- 
ern look on the modes of interaction between sound and movement in Variations V 
could deduct that the artists were not able to implement a precise control of sound 
through movement due to the limitations of the relatively crude analogue system in 
interpreting movement into actuating commands. Nowadays, interpreting movement 
into digital data has given rise to GRT devices that allow a detailed level of control- 
ling software through physical movement. Such systems are implemented in diverse 
applications within the fields of computing, manufacturing, as well as arts. On the 
latter field, researchers and practitioners exploring interaction between sound and 
movement have formed dedicated communities, such as the annual conference New 
Instruments for Musical Expression (NIME). 

While much of the focus of this research field has been the development of new 
technologies and methodologies facilitating the expression of sound through move- 
ment, implementing these contributions demands a certain level of technical profi- 
ciency and knowledge of the associated language and vocabulary. In my practice, 
I was faced with this issue during my early collaborations with Roberts. To further 
contextualize the aforementioned independence between our respective material, our 
approach resulted from the lack of a shared vocabulary. In other words, while Vari- 
ations V presupposed a degree of freedom between sound and movement due to 
imprecise technology, that freedom in my work with Roberts was dictated due to 
imprecise language. Although my mapping of the Wiimotes’ motion data to sound 
processing parameters made sense to practitioners familiar with audio software, com- 
municating these mappings verbatim to my collaborator was inefficient in allowing 
Roberts to understand how her movement affected sound, an issue further com- 
pounded by my own lack of knowledge in the language of choreography. While 
we were able to overcome these issues due to our prolonged collaboration and per- 
sonal familiarity, I was aware that the privilege of time and patience will not always 
be present. This realization prompted me to explore a way of communicating my 
desired intentions in shaping sound through movement via the use of GRT. Prior to 
this specific research question, my research contributed a framework comprising of 
a set of strategies and precepts towards guiding polydisciplinary collaborative pro- 
cess [2], with its core concept being the biological phenomenon of symbiosis. From 
its serendipitous appropriation as the name of our initial collaborative engagement, 
examining the scientific research of the phenomenon in the context of artistic practice 
allowed for the development of a system derived from a disciplinary-neutral field, 
thus avoiding presupposing greater importance to neither of our disciplines. 


8.3 The Symbiotic Phenomenon 


The lexicological definition of symbiosis (Oxford English Dictionary) suggests two 
elements existing in a sustained harmonious relationship. However, in the con- 
text of biological associations, harmonious coexistence is merely one of the many 
manifestations of symbiotic relationships. Biologists define symbiosis as the close 
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and persistent relationships between organisms of different species [8]. Organisms 
engaged in symbiosis are identified as the host and its symbiont, with the engagement 
most often initiated by the symbiont becoming attached to the typically larger host, 
motivated by the former’s desire to extract benefit from the relationship. In line with 
the different levels of extraction of benefit, or fitness outcome [9], symbiosis mani- 
fests in a variety of types. The three core types are mutualism, commensalism, and 
parasitism, with each type identified according to the effects it has on the engaged 
organisms. And since the symbiont is always benefited, it is the effect on the host 
which defines the type (see Table 8.1). 

This taxonomy of relationships refers to symbioses as they are observed at a 
specific moment in time, described as research on ecological scale [10]. However, 
examining symbiosis over longer timescales reveals its function as an evolutionary 
mechanism, with the organisms’ evolutionary trajectory reacting to the close and 
persistent interactions with their partner’s specific traits. For example, while a par- 
ticular partnership may have begun as parasitic (which is often the origin of most 
interspecific associations [9]), the host will eventually evolve ways to extract benefit 
from the relationship, while the symbiont will also manage its exploitation of the 
host with the aim of prolonging the relationship. This reactive evolution serves as 
proof for an evolutionary trend from parasitism to mutualism, which is nowadays 
widely accepted by biologists [10]. 

Having explained the core taxonomy of symbiotic relationships, the next step 
towards drawing parallels between biological and creative associations is to firstly 
identify the elements making up each partnership, and secondly establish a rela- 
tionship between these elements. The first shared aspect among the two forms of 
association concerns the partners’ motivation towards establishing a relationship; 
that is to combine their individual traits as means of jointly overcoming limitations, 
respectively borne of environmental [8] and disciplinary [11] factors. From this start- 
ing point, the remaining elements are placed through a process of conceptual debate, 
and finally become organized opposite each other, as summarized in Table 8.2. 

This subjective interpretation assumes the symbiotic relationship as the collabo- 
rative engagement, with the organisms as the collaborating practitioners. The inter- 
specificity of the engaged organisms is reflected through the different disciplines 
employed by each practitioner, with each discipline’s specific expressive media 
related to the biological traits carried by each species. In interpreting the roles of 
host and symbiont, these are allocated respectively on the practitioner instigating 
the collaboration and the one who is guided according to the former’s direction. As 
will become clearer later on, the used nomenclature applies to engagements which 


Table 8.1 Symbiosis 


Type of interaction Fitness outcome 
typology and fitness outcome 
Symbiont Host 
Mutualistic Positive Positive 
Commensalistic Positive Neutral 
Parasitic Positive Negative 
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Table 8.2 Related elements of symbiotic relationships and polydisciplinary collaborations 


Symbiotic relationships Polydisciplinary collaborations 
Organisms Symbiont Instigator Practitioners 
Host Directee 
Interspecificity Biological traits Expressive media Polydisciplinarity 
Fitness outcome Expressive range 
Fig. 8.2 Effect direction in host <= symbiont mutualism 
the different types of oi . j 
symbiosis and collaborative HE host «= symbiont | commensalism 
m6 


process host == symbiont parasitism 


collaborative | instigator == directee 


Range 


interactive | instigator == directee 


Expressive 


directive | instigator == directee 


feature a higher level of dependency, be that a parasitic relationship in symbiosis or 
a directive collaboration [12] (see note 2) in artistic practice, with the prescriptive 
meaning of these roles diffused during commensalistic/interactive engagements, and 
almost entirely absent in mutualistic relationships and collaborative modes. 

The final interpretation concerns the element of fitness outcome into expressive 
range. As mentioned, the types of symbiotic relationships are identified according to 
the host’s fitness outcome, or the level of benefit that organism experiences as a result 
of its engagement with the symbiont. In the context of my interpretation, expressive 
range refers to the level of creative input allocated to each practitioner during their 
collaboration (see note 3). Figure 8.2 showcases the shared direction of effect between 
the two types of association, with the partner’ extraction (or reciprocal exchange) of 
fitness outcome in symbioses, and delimitation (or conversation) of expressive range 
in collaborations. In the context of polydisciplinary collaborations, this approach 
presents an efficient way of understanding and allocating each practitioner’s liberty 
in developing their respective disciplinary material during the process of collaborative 
engagements (see note 4), with the same principle available towards organizing the 
relationship between musician and dancer while operating an interactive system. 


8.4 Interactive Taxonomy: Theory and Precedents 


As mentioned in the introduction, the typical approach of using GRT in music- 
dance interaction concerns the change of sound through movement. This effect is 
achieved by mapping movement data to various parameters of DSP devices, with the 
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sound consequently affected as a result of the movement data performing alterna- 
tions on the parameter’s values. Considering this relationship between the two media, 
sound can be understood as the symbiont medium, with movement being the host 
governing the development of sound. With this principle in place, and taking into 
account a host’s different fitness outcomes during each type of symbiosis, an equal 
number of interaction modes can be derived, where the ‘host’ movement can be “ben- 
efited’, ‘harmed’, or ‘unaffected’ by its ‘symbiont’ medium of sound. In line with 
the previous subjective interpretation between the elements making up each associ- 
ation, the biological notions describing changes in fitness outcome are related to the 
restrictions, or lack of, placed on the expressive range of the associated media and 
their respective practitioners during a performance. The most efficient way to estab- 
lish these relationships is by first observing the resulting sonic outcomes, followed 
by the restrictions placed on the movement, the provision in which the two perform- 
ers develop their respective sonic and movement material, and finally the dancer’s 
awareness of how their movement affects sound while operating the system. 

Looking at the first association, when the collaborative performance requires a 
determined sonic outcome (akin to a fixed music score), the dancer must perform 
a specific set of movements in order to alter the values of the DSP parameters in 
a predefined manner. As a result, this interaction mode imposes restrictions on the 
movement’s range of expression in order to accommodate the desired sonic outcomes. 
Furthermore, with the mappings between movement data and DSP parameters hav- 
ing been created by the music practitioner, she or he needs to communicate to the 
dancer the required movements needed to achieve the determined development of 
sound over the duration of the performance. Consequently, the dancer is relieved 
from having to fully understand the ways her or his movements may affect sound 
beyond the predefined movements. As such, this interaction mode assumes move- 
ment as a predefined modulator for the sound. In other words, through the previously 
discussed subjective interpretation of the biological notions describing fitness out- 
come, the ‘host’ movement is ‘harmed’ in order to ‘benefit’ its ‘symbiont’ sound, 
thus establishing a parasitic symbiosis between the two expressive media. 

On the opposite spectrum of sonic outcome, an indeterminate score entirely alle- 
viates any requirement for the dancer to become familiar with the mappings between 
movement data and DSP parameters, with movement remaining independent to 
sound. However, from the musician’s perspective, the mapping must be designed 
as to accommodate the dancer’s full range of movements which she or he may 
deploy in any manner and time throughout the duration of the performance. In a way, 
the randomized alteration of DSP parameters during this interaction mode can be 
related to a generative music system, or to provide a further simplified reflection, to 
the modulations derived by an LFO set to random or noise waveform. In the context 
of the symbiotic interpretation, the ‘host’ movement is ‘unaffected’ due to enjoying 
a full range of expression, while the ‘symbiont’ sound extracts ‘benefit’ in the form 
of randomized modulations that can be used to develop and expand its outcome. As 
such, this interaction mode forms a commensalistic symbiosis between sound and 
movement, which on reflection can be associated with the interaction employed in 
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Variations V, when dancers had an effect on sound despite being unaware of the 
ways their movements specifically controlled Cage’s tape players. However, while 
Cage revelled in employed indeterminacy as a compositional approach, Cunningham 
directed his dancers through an explicit choreography. Nevertheless, this is but one 
manifestation of a commensalistic interaction mode, and as I present later in this 
chapter, free improvization presents itself as fruitful provision for the dancer to fol- 
low, with the musician tasked with designing a system able to generate meaningful 
sonic outcome through random modulation inputs. 

With mutualism being the remaining type of symbiotic relationship, such an inter- 
pretation into the context of collaborative performance requires for both sound and 
movement to mutually extract ‘benefit’ from their interaction, which considering 
the earlier connection between fitness outcome and expressive range, suggest a 
simultaneously full range of expression for both media. While the provision of free 
improvization may at first appear salient to this mode, developing this mode through 
practice showcased that an intermediate provision is more appropriate, that of struc- 
tured improvization. Examining this provision in the context of music performance, 
structured improvization differs from its free counterpart by the approach of creating 
real-time compositions by connecting pre-established material over an arrangement 
which is not predefined [13]. As such, while the resulting sonic outcome is not 
determined, its characteristics can be anticipated. Structured improvization shares 
a slightly different meaning in the context of choreography, with dancers adhering 
to a predefined temporal arrangement in relation to stage placement and clustering, 
while retaining freedom towards their performed movements during each section 
of the arrangement [14]. Considering these provisions for music and dance respec- 
tively, in the context of GRT-facilitated interaction the dancer is allocated freedom 
towards her or his movements, with the caveat that these movements need to result to 
anticipated sonic outcomes. As such, the dancer must be well-familiarized with the 
system’s mapping, and be aware of the ways each movement may affect sound. In 
other words, the mutualistic interaction mode presents a mutual compromise between 
the expressive ranges allocated to sound and movement, with both media mutually 
extracting ‘benefit’ up to the level at which one of them can be said to experi- 
ence ‘harm’, thus resembling the mechanisms by which mutualistic symbioses are 
developed over evolutionary scale in the natural world. 

The three symbiotic modes of interaction are summarized in Table 8.3, with 
each mode identified according to their specific effect awareness, provision, 


Table 8.3 Taxonomy of symbiotic interaction modes with associated elements 


Interaction modes 


Mutualism Commensalism Parasitism 
Affect awareness | High Low Moderate 
Provision Structured improvisation | Free improvisation | Score/choreography 
Operation Exploration Detachment Instruction 
Outcome Anticipation Indeterminacy Determinacy 


130 M. Moriaty 


operation, and outcome borne of the interaction between the two media. At this 
stage, it is worth pointing out once again the subjectivity and conceptual nature of 
this interpretation, and furthermore the Practice Research methodology employed 
towards reaching these findings, with the latter derived through the accumulated 
knowledge from numerous years of collaborative practice alongside several prac- 
titioners. As presented later in this chapter through the work Symbiont Zero [15], 
in addition to employing distinct modes of interaction, multiple modes can also 
manifest during a performance, either consecutively during different sections, or 
simultaneously while operating different layers of sound, each controlled via a dif- 
ferent mode. Prior to describing our collaborative outcome and designed interaction 
system, I demonstrate the taxonomy’s theoretical standing by examining three inter- 
active works as means of identifying their specific modes of interaction. Rather than 
providing a conclusive literature review of the field’s practical outcomes, the pre- 
sented precedents were selected as to represent distinct uses of GRT, collaborative 
processes, and employed aesthetics. 


8.5 Symbiotic Interactions in Precedent Practices 


Stratofyzika is a Berlin-based collective, with its founding member comprising musi- 
cian Lenka Kocisova, visual artist Alessandra Leone, and dancer Heather Nicole (Hen 
Lovely Bird). Their 2016 work THATA [16] presents a dialogue between the three 
media of sound, movement, and visual. In the context of the symbiotic concept, the 
dancer serves as the host modulating sound and visuals, with the interaction facil- 
itated through a wearable Inertia Measurement Unit attached on the dancer’s arm, 
with the movement data simultaneously modulating the two software suites gener- 
ating sound and visuals. According to Leone, rather than having a constant flow of 
movement data into the system, this flow is interrupted by altering the correlation 
between movement data and resulting value alteration via algorithms [17], as well as 
activating the data stream only during predefined moments. With the dancer aware 
of the way her movement affect the other two media, it is during those moments of 
activated interaction that the dancer is able to freely employ movements that result 
anticipated sonic and visual outcome, such as the floor roll (see [16], 1:10-1:20) 
resulting in a white spire and a tone of descending pitch. 

The second precedent is one of the audio-visual installations developed with 
the danceroom Spectroscopy (dS) visualization system. Using an array of depth 
perception cameras (Microsoft Kinect) as its GRT devices, the system is able to 
extract movement data from multiple participants, with the date used to modulate 
the visualizations. While dS was designed primarily as a research tool for molecular 
dynamics, allowing researchers to visualize particle movement, as well as interact 
with their visual representations [18], it then became the basis for a series of creative 
outcomes with the addition of a sound-generating system sharing the data modulating 
the visualizations, with the movement data coming from a troupe of dancers. In 
addition to the performances, the team behind dS also presented the system in the 
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configuration of a public multi-participatory installation, where audience members 
are invited to interact with the system. The result of this interaction is that sonic and 
visual events are manipulated by movement without the controllers having any prior 
training with the system [19], thus demonstrating a commensalistic interaction mode. 
A point of interest here is that once an audience member began exploring the system, 
they would attempt to understand what their effect is on the media, resulting in an 
adaption of the interaction mode. While this does not in itself form an undesirable 
aspect, I later present a solution devised towards maintaining commensalism during 
interaction. 

Finally, as an example of practices displaying parasitic interaction between sound 
and movement, Marco Donnarumma’s Corpus Nil [20] is a work which appears to 
absolutely appropriate the notion. For this work, Donnarumma devised a choreog- 
raphy comprising of ‘five key bodily postures’ [21], each designed as to force the 
system to generate a specific response from the sound and lights it controls. With 
the system operating through machine learning algorithms able to identify specific 
positions by their duration, the performer must sustain these positions as to progress 
through the arrangement. As such, Donnarumma is restricted by the design of these 
key positions, or more accurately, is able to move only within the restrictions posed 
by each position. Understanding this approach through the symbiotic framework, the 
performer possesses a detailed awareness of the ways their movement affects sound 
and lights, and is able to perform the determinate sonic outcome through their move- 
ments. This sits in contrast to the approach employed by Stratofyzika and danceroom 
Spectroscopy; in the latter, audience members are unaware of the ways the system 
interprets their movements as modulations for the other two media, whereas in the 
former, the dancer is aware of the effects her movements have on the interacting 
media, and is allowed to explore movements along with the effects these have on 
sound and visuals. In the case of Corpus Nil, the performer’s movements are intended 
to achieve predetermined changes to the other two media. 


8.6 Symbiotic Modes of Interaction in Symbiont Zero 


As mentioned earlier, the findings presented in this chapter stem from my collabora- 
tive practice alongside several dancers. Following the development of the theoretical 
frameworks for polydisciplinary collaboration and symbiotic modes of interaction, 
my aim was to activate the entire spectrum of interaction modes within a single 
creative outcome, with the additional purpose of serving as a demonstration of the 
framework. In order to satisfy both purposes, Symbiont Zero has been documented 
both as complete performance, as well as a set of demonstration videos, with the latter 
displaying a simplified version of each interaction mode, with additional perspectives 
providing simultaneous views of the musician’s inputs via external controllers, as 
well as the effects both performers conduct on the mapped parameters (see Fig. 8.3 
for a representative example). 
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Drop Audio Effects Here 


Fig. 8.3 Demonstration video with multiple views: dancer (Cunliffe), external controller, and 
mapping distribution matrix 


The work was initially developed with my long-term collaborator Roberts, and 
subsequently performed with contemporary dancers Lucie Lee, Joseph Lau, and 
more recently Kelsea-Leigh Cunliffe. The premise concerns a musician-dancer duet 
featuring interaction between their respective expressive media through GRT. With a 
total duration of approximately fifteen minutes, the piece develops over three move- 
ments, or sections, of equal duration, and while each section predominantly focuses 
on a single mode of interaction, the performers have gone to implement areas of 
adaptation through simultaneous employment of different modes while controlling 
multiple layer of sound. 

In describing the path of the movement data into the system, the concerned GRT 
devices comprise a pair of Wiimotes tethered on the dancer’s forearms as means of 
performing alterations to the sound-generating system. This system is hosted within 
Ableton Live, receiving a total of eight data streams from the two Wiimotes, follow- 
ing their capture and conversion into MIDI CC messages through the application 
OSCulator. These continuous messages are then mapped onto two Max for Live 
devices, namely, Map8 and Multimap, which serve two purposes; firstly, acting as a 
distribution matrix from which to map each data stream onto multiple DSP param- 
eters, and secondly, allowing the musician to toggle the activity of each mapping, 
as well as alter the range, direction, and curve of each mapping’s effect on their 
destination during the performance. The musician’s inputs are achieved through two 
hardware MIDI controllers combining continuous and toggle controls. The data path 
is graphically represented in Fig. 8.4, (albeit for the data from a single Wiimote 
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Fig. 8.4 Path of gesture data, from dancer’s movement to DSP mapping 


only), with the following subsections describing the work’s three sections in terms 
of both system operation and performance planning. Each section is described under 
the title of their predominant mode of interaction. 


8.7 Mutualism 


In line with the aspects of mutualistic interactions, the dancer has been familiarized 
with the system, and is able to anticipate the ways her movements may affect sound. 
Furthermore, the performers have noted certain combinations of movements and 
specific ranges in the distribution matrix which resulted in interesting sonic outcomes, 
which the performer has memorized and able to deploy at will. 

The main interactive sound during the mutualistic section is generated by a feed- 
back delay (Amazing Noises Dedalus Delay), with its parameters mapped to the 
Wiimotes’ eight continuous messages, these being X-axis (vertical position), Y-axis 
(roll), Z-axis (yaw), and acceleration. In this case, the X-axis of both devices is 
assigned to the delay’s two filters, low pass and high pass for left and right arm, 
respectively. This particular mapping ensures that the performer maintains control 
over the overall signal amplitude, which subsides by simultaneously pointing both 
arms upwards due to the removal of high or low frequencies by either filters. The 
remaining mappings are arranged to parameters such as pitch shifting, size and 
frequency of grain, and addition of parallel distortion and reverberation. 

The musician’s role during this section is to adjust the range by which each con- 
tinuous message affects the mapped parameters. This is achieved by adjusting the 
range parameters on the Map8 device, thus altering the dancer’s level of influence 
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on each mapped parameter. The section begins with the filter ranges restricted mid- 
dle frequencies, while the dancer concentrates on upper body movements. Halfway 
through the section, the musician broadens the range of the filter mapping, causing a 
sudden increase in amplitude. This serves as a signal for the dancer to initiate weight 
transfer movements, resulting longer movements across the stage. 


8.8 Commensalism 


As described in previous sections, commensalistic interactions alleviate the need for 
the dancer to possess prior knowledge of the ways movement affects sound. However, 
as mentioned in the description of dS, participants in control of sound will naturally 
begin to discover the connections between their movements and changes in sound, 
resulting in an adaptation of mode. Aiming to avoid this adaptation during the com- 
mensalistic section of Symbiont Zero, I designed a system able to dynamically alter 
the mapping during the performance. This is achieved by mapping a single move- 
ment stream of continuous change message to several DSP parameters by inserting 
an instance of Multimap between one of Map8’s outputs and the DSP inputs. Fur- 
thermore, toggles are assigned to toggle each of the Multimap’s outputs. The result 
of this design allows the musician to continuously alter the movement data stream 
distribution among the mapped DSP parameters, and thus avoid the dancer to gain a 
fixed relationship between her movement and specific sound modulations. 

The sound generation is based on a series of granulators (Audiority Grainspace) 
processing recordings of synthesized percussive patterns, with the Wiimotes’ CCs 
mapped to parameters such as start—end of input sample, pitch and tone variations, 
saturation, size of and distance between grains, and level of parallel reverberation. 

While the dancer is entirely at liberty to freely improvise her movements, she 
is also motivated to ‘forget’ the interaction, and instead treat the resulting sound as 
recording, akin to improvising to a fixed (non-interactive) composition. The only 
cognisant interaction of the performer occurs after a new layer of sound appears, 
made of two percussive samples (respectively kick drum on the left arm and snare 
drum on the right arm), each activated by the acceleration streams going over a set 
threshold which triggers a MIDI Note on message mapped to an instance of Ableton’s 
Drum Rack containing the percussive samples. 

Once the performer becomes aware of the new layer, she is able to interact with 
the two sounds through a mutualistic mode, while simultaneously maintaining her 
commensalistic interaction with the previous sound layer. The section ends with the 
initiation of a further sound layer made of arpeggiated synthesiser pattern, signifying 
the transition into the final section. 
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8.9 Parasitism 


The parasitic interaction presents the highest level of accuracy from the dancer, while 
at the same time imposes the highest level of restriction on their movements. The 
Wiimotes’ CCs are mapped to a synthesiser (Ableton Analog) generating a fixed 
arpeggio pattern, with the left arm controlling the synthesiser’s low pass filter cutoff 
frequency on the X-axis and envelope attack duration on the Y-axis (rotation), while 
the right arm controls reverb depth and delay feedback on X- and Y-axis, respectively. 
In line with the approach during parasitic interaction modes, the sonic outcome is 
determined, in this case being a slow modulation between the wet-dry balance of 
the reverb, followed by a filter frequency sweep of equal duration. At the same time, 
the determined outcome requires for staccato (sudden jumps in value) modulations 
on the two remaining parameters, resulting in altering the note’s envelope shape and 
feedback level. Finally, the two percussive samples from the previous sections are 
required to be activated at the end of each filter sweep cycle, resulting in the dancer 
interacting with this layer through a parasitic mode, as opposed to the previous 
mutualistic engagement. 

With the determined sonic outcome providing a limited range of movements for 
the dancer, the latter is then free to develop their own choreography within this 
restricted range. Having performed the piece with four different dancers, a partic- 
ularly interesting finding is that while each dancer were able to achieve the deter- 
mined sonic outcome, each reached that through distinct sequence of movements. 
For example, Roberts maintained her core still while only moving her arms (as to 
achieve the modulations on the synthesiser’s parameters), Lee continued to move 
across the stage while limiting the movement of her arms in line with the score. 
Similarly, while Lee activated the percussive samples by performing waving move- 
ments with her arms, Cunliffe opted to perform sudden contractions and extension, 
akin to punching motions. This showcases that despite the imposed restrictions dur- 
ing parasitic interactions, dancers are able to interpret the score through different 
choreographic movements. 

In describing the remaining section, once the filter sweep sequence has been 
performed twice, the dancer is instructed to resume a mutualistic interaction with the 
percussive samples, while at the same time the musician limits the mapping range of 
the synthesiser parameters, who is then able to assume control of these parameters 
through their external controller. As a result, the musician ‘takes’ away control from 
the dancer while she breaks free from the previous restrictions. This continues for 
around one minute, before the dancer assumes the previous position, and the musician 
returns the control of the synthesiser back to the dancer, who repeats the previous 
movements twice. Symbiont Zero concludes with a short section of commensalistic 
interaction, where the performer’s movements influence a pair of granulators (New 
Sonic Arts Granite) processing two segments of recorded music, namely, the ‘Amen’ 
and ’Hot Pants’ drum breaks. As with the previous section, the performer is unaware 
of the effect their movements exert on the newly introduced sounds, and is instructed 
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to perform an improvized sequence, with its intensity influenced by the overall level 
of sound made of the combined elements, which begins to fade until silence to the 
end of the piece. 


8.10 Conclusions 


Although the analyzed precedents do not constitute an exhaustive review of the field, 
the interactive works portray a selected collection of practices displaying diversity 
in GRT (Inertia Measurement Units, depth perception cameras, and biophysical sen- 
sors), in the make up of their collective creation (inter/multi/trans-disciplinary respec- 
tively), and as explained earlier, in the employed interaction mode between physical 
movement and digital audio-visual material. Appropriating the typology of symbi- 
otic relationships towards forming a new taxonomy in technologically facilitated 
interaction allows for the diversity in which interactive performances manifest to be 
identified within just three modes of interaction, each comprising of thee aspects— 
provision, operation, and outcome—as well as the level of training or familiarity with 
the system required by the dancer. This serves as an efficient approach towards com- 
municating the purpose of each mode, particularly assisted by the lack of convoluted 
terminology, which can often be counterproductive when used to communicate the 
desired outcome to practitioners whose expertise focus on other areas of practice. 

This research has thus far focused on dyadic collaboration, with future view of 
assessing the symbiotic concept with multiple partners. Furthermore, while the use of 
consumer GRT devices was implemented as means of making the research outcomes 
more inclusive (due to being both reliable and inexpensive), the limited accuracy and 
expansion of these devices have become to emerge in my practice, which serves as 
motivation to look into developing bespoke devices, in line with current trends in the 
field of interactive arts research. 

Most importantly, however, it is my strong belief that the future of interactive 
arts will greatly benefit from researchers combining technological innovation with 
artistic Practice Research, a symbiotic relationship if you will. In my experience, 
although some steps are already taken by few brave individuals, the two fields’ distinct 
approaches, cultures, sensitivities, and nomenclature result in hesitation towards 
tighter integration. But akin to the parasitic beginnings from which most symbioses 
begin their evolutionary journey, persistence and closeness will eventually manifest 
in mutual aid among the two domains of creative practice, artistic and scientific. In 
the words of Pëtr Kropotkin, ‘In the practice of mutual aid... we thus find the positive 
and undoubted origin of our ethical conceptions; we also see the best guarantee of a 
still loftier evolution of our race’ [1]. 
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Notes 


(1) 


The term polydisciplinary is used to denote the multiple modes by which disci- 
plines interact, as an alternative to the more popular ‘interdisciplinary’, which I 
consider to be erroneously habituated for this purpose, as it ignores established 
principles of taxonomy. I provide a more thorough discussion on this subject in 
Sect. 2.2.2 of [3]. 


(2) The referenced modes of collaboration are derived from the work of Hayden 
and Windsor [12], identified as directive, interactive, and collaborative, each 
suggesting a different level of creative liberty between the engaged practitioners. 

(3) The social aspects of collaboration, along with the notions of allocating creative 
direction duties, privileges, and responsibilities, have been addressed in my 
previous paper [2]. 

(4) Fora detailed description of the symbiotic framework on collaborative process, 
see chapter three of [3]. 
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Chapter 9 R) 
Between Presence and Program: The geai 
Photographic Error as Counter Culture 


Tracy Piper-Wright 


Abstract Common photographic errors such as over or under exposure, blur, or 
inadvertent cropping are increasingly rare as technological developments in digital 
photography have sought to eradicate the error from practice and perception. Effi- 
ciencies such as camera automation and image preview are often designed to remove 
the ‘unreliability’ of the human element in order to produce accurate and consis- 
tent images. The error, occurring on the margins of practice and increasingly rare, 
provides a counterpoint to this prevailing image culture by revealing the interde- 
pendence of photographer and camera through unintended outcomes. This chapter 
explores the ideological assumptions entwined in the development of camera tech- 
nologies, and how cultures of practice based on a hierarchy of control between 
camera and photographer arose. Through examples drawn from the research project 
In Pursuit of Error, the chapter demonstrates how the error disrupts this hierarchy by 
evidencing the shared subjective agency of camera and photographer. The method- 
ological framework of Actor-Network Theory is used to interrogate the relationship 
between photographer and camera and to reveal them as equal ‘actants’ in the event of 
photographing. The embodied photographer’s attitude of play, experimentation, and 
not-knowing is interdependent on the camera as a co-creator of unexpected image 
events which disrupt the conventions of photographic representation. 


Keywords Photography - Error - Agency - Doubt - Actor-Network - 
Embodiment - Subjectivity - Time - Hierarchy of agency - Digitization - 
Virtuality - Actor-Network Theory - Transformative art - Alternative realities - 
Presence » Cultures of practice - Experimental photography 


9.1 Introduction: Histories and Hierarchies 


Alone of all the creative arts, photography is the most deeply enmeshed with the 
technologies of production. A photograph cannot be made without an instrument 
external to the photographer whereas one might argue the technologies of drawing 
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or painting can, on occasion, be replaced by the human hand and the right surface. 
The symbiosis between camera and photographer makes for a peculiar push and 
pull between the claims of the technical and the subjective, which have been the 
hallmark of writing about photography as, variously, an art, a craft and a science 
since the inception of the medium. The tension between an apparently objective and 
automatic picturing mechanism and the role of the photographer in the creation of 
the image dates back to the critics and practitioners of the late nineteenth century, 
and centers around a hierarchy of agency—who or what is finally responsible for the 
photograph? 

The unavoidable connectedness between camera and photographer is not a fault 
of the practice, but rather an important distinction which must be recognized in any 
discussions about it. In the early twenty-first century the digitization of photography 
coincided with the connectivity offered by the Internet, producing the circumstances 
for limitless digital images to be created and subsequently shared and viewed in 
distributed networks. This rapid development created extra layers of technological 
distance between photographer and camera as points of tension emerged between 
the physicality of analog film and the ‘virtuality’ of the digital photograph [1, p. 97]. 
As digital cameras become increasingly abstruse and proliferate the question of the 
relationship between it and the photographer becomes one of knowledge as well as 
agency. If I cannot know what the camera is actually doing, how much control can I 
have? The “black box” [2, p. 27] of the digital camera lies so far outside the realms 
of ordinary understanding that photography has almost returned to the realm of the 
magical. 

What made photography ‘magical’, in its earliest presentations, was its ability to 
capture and retain an image of the world with a precision hitherto impossible through 
the human hand alone [3]. While early discussions on the invention of photography 
refer to the photographic process rather than the camera itself; the technology of the 
camera as an anthropomorphized ‘eye’ was first proposed by Da Vinci in relation 
to the camera obscura [4] the earliest form of enclosed light projection from which 
cameras were developed. Thus, some assumptions about photography were set out at 
the early stage of its development: a monocular, perspectival viewpoint which mimics 
the focus of attention in human vision and a unique responsibility to reality which 
is the product of an “unreasoning machine” that produces “an unerring record” [5, 
p. 269]. 

The camera’s ‘eye’ supersedes the frailties of the human eye through its objective 
stance and ability to transmit and reveal the world as it is, without interpretation 
or editing. The camera becomes a technology for asserting truth, and the images it 
produces are facts that can be used as evidence. 

Leading from its perceived status as a truth-telling machine, the quest for sim- 
plified and accessible accuracy has been a dominant motivator in the technological 
development of the camera. The famous Kodak strap line which accompanied the 
launch of its camera in 1888 “You push the button - we do the rest” [6, p. 38] exem- 
plified a desire to remove the possibility of error in the creation of a photograph by 
eliminating the more complicated or technical aspects both in camera design (the 
camera had a fixed focus lens and limited aperture/shutter speed controls) and in 


9 Between Presence and Program: The Photographic Error ... 141 


the loading and development of film (the whole camera was sent back to Kodak for 
processing, thence replenished and returned to the owner) [6]. 

The Kodak camera is evidence of the twin aims of technological progress which 
can be found in most subsequent camera development including digital. Firstly, there 
is the desire to ameliorate the complexity of the photographic process per se; the 
unknowable chemical/algorithmic functions which are performed in the making of 
the image. Secondly, the camera sets out to moderate the potential of a faulty, error- 
prone human element from entering the photographic equation. The photographer is 
reduced to being a button pusher, and no more. The twin aims of camera development: 
simplicity and error elimination can be seen in many of the innovations in camera 
technology in the twentieth century, such as cassette film and auto-focus, which 
continued into the twenty-first century digital revolution. 

These reflections on the early history of photography show that the pursuit of 
accuracy has been at the forefront of the practice and perception of photography since 
the beginning. The linking of photography as a technical yet simplified image making 
method that requires the presence of a ‘button pusher’ but which devolves the image 
making to the camera, makes for a peculiar awkwardness in the relationship between 
human and camera: an oddly mismatched hierarchy in which the photographer is 
perceived as both necessary and extraneous to the photographic event. 

Despite these advances, errors still crept into the practice of photography. Not 
every eventuality could be eliminated, and the material agency of the film and the 
chemical process contained many potential pitfalls for the photographer, which were 
often only revealed once the prints had returned from the developer. Unsatisfactory 
images could be discarded, but were often simply incorporated back into the wallet 
of photographs. The gap between the making and revealing of an error effected a 
nostalgic distance between event and object in which the significance of the image 
could override the obvious flaws in the depiction. In the analog era photographs were 
perceived as a finite resource, and an error might appear a charming reminder of a 
moment in time. 

Digital photography took its cues from film photography in terms of camera 
design and it is therefore unsurprising that its subsequent development extended 
the simplicity/error-elimination methodology. However, digital photography offered 
a level of certainty that far exceeded what had been possible before. Due to the 
algorithmic nature of the sensor and digital processor cameras became computers, 
and the photographer became less and less the primary decision-maker [7]. Later 
developments such as image preview effectively capped the potential for error by 
making it possible to delete substandard images at source, narrowing the possible 
images that could emerge from any given shoot. In the digital era the opportunities 
for ‘casual’ error creation changed; no longer could a substandard image slip through 
the net and into the pile of prints. Each image could be vetted so that only the best 
were eventually presented in online spaces. 

As our consumption of photography as digital images in networked environments 
increases the more our perception of photography reflects what we are presented 
with—invariably ‘good’ photographs. In this case a ‘good’ photograph does all that 
a photograph should: it shows things as they are, to the best of the camera’s ability. 
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The technology of the camera, now most often embedded in a phone, can control 
lighting, saturation, and shadows to produce high quality images that feel ‘just like 
being there’. This quest for transparency, for a feeling that you are looking at or 
experiencing the ‘real thing’, has been the ultimate goal of consumer digital photog- 
raphy, with increased mega-pixels offering more and better rendering of the scene. 
The photograph becomes a window through which we observe the phenomenon 
depicted. This hyper-verisimilitude removes even the camera as the creator of the 
work, ostensibly rendered out of existence by the perfection of its image [7]. 

Concomitant with these advances is the increase in the quantity of images made 
available by digital technology. Not only do the best arrive on the Internet, but pretty 
much everything else, so that we wade through a sea of images at every turn. While 
photographs enter the network within a particular context they do not remain there 
in perpetuity, and can soon break free to circulate and appear in multiple sites and 
locations. The plurality and mobility of the digital photograph leads to a dissociation 
of the image from its context in time and place and a peculiar sense of authorlessness, 
where even the fact that we are looking at an image made by a camera pointed by a 
human can escape our perception. 

The foregoing commentary sets out the context in which the research project In 
Pursuit of Error [8] (hereafter referred to as IPE) emerged and seeks to challenge. 
The project collects photographic errors from artists, photographers, and the general 
public in order to form an archive of the error in contemporary practice. Contributors 
are asked to include a statement that explains how their photograph occurred and 
why they feel it is an error. Through analysis of the images and narratives, the project 
reveals how artists value the accident as a point of discovery and development and 
engage in playing with and against the camera in the creation of ‘deliberate’ errors. 
The project aims to question some of the assumptions about representation and 
truth which beset photography as a practice, and to challenge the assumption that 
technological developments manifest objectively rather than because of cultural and 
ideological assumptions about what is correct or good. Thus the project attempts 
to destabilize the hierarchy that attaches to the relationship between camera and 
photographer (with one or the other ‘in control’ at any given time) in favor of exposing 
the horizontality of this relationship [9, p. 283]. The project uses the error to explore 
arevitalized concept of photography as an embodied, performative practice, in which 
the bodies of photographer and camera play a part. 


9.2 Camera Acts and Human Acts 


In order to consider how humans and cameras interact in the creation of a photo- 
graph, it is necessary to think through the processes that bring a photograph into 
being. Despite common assumptions about time in relation to photography—the 
split second “capture’—making a photograph has a number of stages, from the first 
desire to take a photo, to preparing or starting the camera, to focusing or selecting the 
composition through viewfinder or on screen and only then the taking of the photo. In 
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this extended time period both camera and photographer perform a number of acts, 
the photographer her gestures, the camera ordered by the principles of its program. 
Within these parameters correct images are to be had; however, flouting the rules of 
either program or gesture tend to produce unexpected or unintended results. 

It is useful to consider the practice of taking a photograph as operating on a 
continuum of control between agency and automatism. While one might assume 
that ‘agency’ pertains solely to the photographer and ‘automatism’ to the camera, 
in practice these roles are interchangeable [10]. Automatic gestures—holding the 
camera steady, or straight, or automatic programs such as auto-focus or aperture 
control create more or less expected results according to the scene presented. Agency 
plays its part in the decisions or mistakes of the photographer but the camera’s 
agency can also intrude on the photographer’s intentions, resulting in surprising and 
unexpected images. 

The photographic errors submitted to IPE cover a range of accidental or unintended 
images and ‘deliberate’ errors in which a willful action has been made to achieve an 
unpredictable result. In many cases it is near impossible to identify from the resulting 
error image which party was the instigator, without the photographer’s identifying 
narrative in support (Fig. 9.1). While the visual distinction between the two types 
of errors is not necessarily apparent, they create a useful distinction which permits 


Fig. 9.1 Image copyright © Gail Griggs and reproduced by permission. Source: In Pursuit of Error 
[8] 
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a focus of attention on either actor in the event of photographing—the camera or 
the photographer. The narratives that accompany the contributions to IPE reveal the 
push and pull between the photographer’s vision and the camera’s vision, and where 
the presence of the camera as agent in the creation of the image is made apparent. 


9.3 Agents and Networks 


To consider the camera as an agent, or an accomplice, in the act of photographing 
is to reconsider the role of the technology and our relationship with it—as user, 
as master, or as collaborator. Actor-Network Theory (ANT) creates a framework 
through which to reconsider this relationship, based upon acknowledgement of the 
“symmetry of humans and non-humans” [10, p. 642] in networks. Networks are 
formed of social practices and technical processes as well as human and non-human 
‘actants’ and vary in scale. A network could be as small as the photographer and her 
camera, or can be broadened out to consider the camera’s developer and manufac- 
turer, the photographer’s educators and peers, and the social and cultural context of 
photography practice. ANT emerged from the sociology of science and technology 
as a means to critique assumptions about the development of technology in soci- 
eties, pointing out relationships of power and questioning deterministic models of 
scientific development which present such development as rational and autonomous. 
ANT?’s non-hierarchical emphasis on ‘actants’ erodes distinctions between human 
and non-human actors and creates opportunities to consider how technology not only 
“speaks back’ to us through our interactions with it, but also how technology governs 
and en-frames our performance with it [11]. 

In the context of Actor-Network Theory the error sheds new light on the rela- 
tionship between camera and photographer. Many of the accidental errors that are 
presented to IPE occur due to the mismatch between the photographer’s intentions 
and the camera’s program, or more strictly speaking, the settings that have been 
selected on the camera. Most modern cameras permit the pre-selection of aperture 
and shutter speed to account for the proximity of the subject and light conditions. 
Errors occur when the conditions fail to match the camera’s settings, or vice versa 
(Fig. 9.2). In this case the situation being photographed becomes another actant in 
the network, a set of conditions which intrudes upon and affects the outcome. 

In the surprise that is created by these unexpected images the photographer is made 
aware of their reduced role in the event of photographing, instead giving over the task 
of picturing to the camera. This mismatch between program and situation, ‘translated’ 
by the camera, presents a wholly new way of perceiving, a form of camera vision 
which reminds us of the subjectivity of perception per se. While gross mismatches 
between program and conditions create image-worlds far beyond our imagination, 
small collisions between settings and situation create spikes of maladjustment in 
the image through which the presence of the camera as an observing body can be 
detected. The accidental error unequivocally brings the camera’s presence to the fore 
(Fig. 9.3). These images do not allow us to look ‘through’ or ‘past’ the photograph 
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Fig. 9.2 Image copyright © 
Ian Wright and reproduced 
by permission. Source: In 
Pursuit of Error [8] 


as medium, instead, through its faults, the ‘photographicness’ of the image is made 
glaringly apparent. The accident is therefore a valuable reminder of the technological 
mediation which the camera brings to the act of photography, a subjective vision 
which is easily overlooked in conventional photographs. 

Photography’s connection to the principles of objectivity and neutrality, which are 
socially and culturally grounded, have accustomed us to treat the camera as a perfect 
imaging-machine. The accident, revealing the fallibility of this conceit, reminds 
us that while automation exists, it is not a panacea which will offer unblemished 
facsimiles of the world every time. The accident demonstrates the camera’s vision 
as subjectively driven rather than an objective fact and foregrounds the extent and 
limitations of the technology that lies behind that vision. These types of errors also 
reveal the role of the photographer in choosing the most appropriate functions to 
create an image, and thus the interdependence of camera and photographer in the 
event of photographing. 

Not all accidents are the result of camera malfunction or wrong settings. Acci- 
dental images also arise through physical mishap (falling, shaking) or something 
inserting itself between the lens and the scene at the moment of capture. These 
intrusions into the photographic event evidence the wider field of actants in the Net- 
work of photographing. While the camera/photographer nexus would appear to be 
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Fig. 9.3 Image copyright © Catalina Codreanu and reproduced by permission. Source: In Pursuit 
of Error [8] 


the sole locus of creation, the error reveals the multiple secondary elements: light, 
time, subject, background; the numerous observable and unobservable phenomena 
that can impinge on the image. The accidental error reveals the momentariness of 
the photograph, its situatedness in time and space and its fixed location and view- 
point. Conventional photography practices strive to present a scene which is as far 
as possible universal, a witness to an event rather than a participant, working at an 
observing distance which lends the scene a sense of objectivity and ‘truth’. This 
photographic stance, as common in newspaper reportage as holiday snaps, relies on 
a detachment between the photographer/camera and the scene, either through pose or 
‘stilling’ of the action (‘smile!’) or through remaining on the periphery. By contrast, 
the accidental error provides incontrovertible evidence of the embeddedness of the 
camera/photographer within the flow of space and time through which pours all that 
is normally left out of the photograph (Fig. 9.4). Through happenstance the error 
records these fleeting intrusions and rapid changes that make up our customary lived 
experience, fixing for a second the swirl of life that we are part of. 

Accidental errors create opportunities to observe the camera at work in the creation 
of the image, in many cases an overlooked or taken for granted act. For Borda [12] the 
physical materiality of the camera is in danger of being left behind in an increasingly 
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Fig. 9.4 Image copyright © Brett Chapman and reproduced by permission. Source: In Pursuit of 
Error [8] 


digitized, automated picturing world which makes the camera “a missing entity that 
we no longer perceive.” [12, p. 179]. The physicality of the camera has been greatly 
diminished by recent developments such as mirror-less technology which removes 
weight and mechanical action and in mobile phones which integrate the camera as a 
part of a hand-held computer. If the ‘objectness’ of the camera is somewhat at risk 
from these developments, the accidental error reasserts it as a form of what Jane 
Bennett would term ‘thing power’: the agency of material objects to act and impact 
other materials, thereby producing effects in the world [13]. The thing-power of the 
camera is writ large in the accidental error, where the photographer is left wondering 
what might have occurred, standing outside the making of the photograph, receiving 
the image as a fait accompli. For Bennett, acknowledging ‘thing power’ entails 
recognizing what she terms its ‘out-side’, that which we know to exist but can only 
indirectly perceive [13]. The camera exercises its ‘thing power’ through the hidden 
process of bringing the photograph into being. We know the theory but the actuality 
of the process is held in abeyance, because the chemistry or the program is closed 
to our perception [2]. This hiatus is where the camera exerts its agency and where 
for a few moments the balance of power between photographer and camera is tipped 
in the camera’s favor. This gap in our knowledge should remind us that we are not 
necessarily masters of our technology, but rather informed collaborators, operating 
on a certain amount of knowledge and an element of trust. 
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9.4 From Tool to Plaything 


The hiatus that occurs in the moment of translation from event to image is an aspect 
of all photography, correct images, and mistakes alike. This loss of control, how- 
ever momentary, seeps into the practice of photography and informs the relationship 
between camera and photographer which transpires to be a balance between knowing 
and not-knowing, of control and loss of control. Thus, photographs could be orga- 
nized on a continuum from those made with complete knowledge and command, 
and those produced as a result of a complete loss of control and ignorance. The error 
could be said to occupy the middle to latter part of the continuum and thus express 
not a simple binary distinction between ‘correct’ or ‘faulty’ images but rather an 
elongated space of practice in which minor to major mistakes or maladjustments, 
made by either party, impact upon the eventual photograph. 

Control could be considered a cornerstone of camera development through techno- 
logical enhancements in efficiency and ease of use. The vagaries of the photographic 
process are brought under control by innovations such as cassette film, the digital 
sensor or ‘Auto’ pre-sets. The control is purportedly for the photographer, who can 
now manage the process more readily, but in fact the control, particularly in relation 
to programmable functions on modern digital cameras, is situated firmly with the 
camera, or perhaps we should say the camera’s manufacturer. Elements of control 
can also be exercised through a culture of photographic practice; the way in which 
one should hold the camera or operate the shutter was often of particular concern 
in manuals for new style cameras where the photographer’s performance with the 
new device was as much under scrutiny as the camera. Flusser points out that both 
‘apparatuses’ (as he terms cameras) and photographers are subject to the control of 
the Program: the systems which define and limit the practice of photography [2]. 
Flusser is rather negative about the possibility of photographers escaping the limi- 
tations of the Program but he does recognize the agency of the individual in testing 
or subverting the expectations of picturing. For Flusser, the increasing functional 
capabilities of the camera, its automation of the photographic process, have liberated 
the photographer from the ‘work’ of making pictures. Now there is nothing else to 
do but play [2, p. 29]. 

Playing with the camera to undermine expected or ‘correct’ image making can be 
seen in the ‘deliberate’ errors, or rather, photographic subversions submitted to IPE. 
These images are often accompanied by a narrative that suggests the photographer’s 
moment of performance with the camera and their action in the moment that has 
brought the image into being. Subversions often involve the photographer’s physical 
movement, running counter to their expected role of stillness and stability—a human 
tripod which facilitates the camera’s task. These movements can be complemented 
by a knowing destabilization of the camera functions—extensions of shutter speed 
to slow down the camera’s capturing eye, forced de-focusing to render the scene into 
amorphous blobs of color (Fig. 9.5). 

These actions could be characterized as playing against the camera, forcing 
it to see in a very different way through manipulation of the controls. Whether 
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Fig. 9.5 Image copyright © Melis Cantürk and reproduced by permission. Source: In Pursuit of 
Error [8] 


through movement or manipulation the subversion foregrounds the photographer as 
an embodied presence in the act of making photographs in a similar way that the 
accident reveals the camera’s subjective viewpoint. The subversion interposes the 
photographer between the camera and the scene as much as if a stray thumb or fore- 
finger were to appear in the frame—we are reminded that the camera is not simply 
alone and recording by itself as an objective observer. The subversion reminds us that 
there is a photographer who is pointing, choosing, moving or otherwise governing 
the activity that precedes the image. 

These subversions become traces of performance in time, capturing a certain con- 
figuration of events that involve camera, photographer, and situation. One may well 
ask why photographers feel compelled to intervene in the process of photograph- 
ing in this way. Certainly, the desire to do so is evident throughout the history of 
photography, notably expressed as a principle of the Provoke photography move- 
ment in Japan in the 1950s in their mantra of ‘are bure boke’ —“grainy, blurry, out 
of focus” [14]. Practices which test the conventions of photography can be seen in 
the work of practitioners as diverse as Meatyard [15] and Barth [16] whose out of 
focus experiments suggest alternative ways of using the camera as a subjective and 
sensory mode of expression rather than an objective recording device. The deliberate 
error is therefore more than just play; it is an urgent reassertion of creativity in the 
act of photography, one that seems to be increasingly marginalized by technological 
advancement which attempts to drive the unreliability of the human presence further 
away from the photographic act. 
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Deliberate errors may also be motivated by a desire to not produce photographs. 
One of the main issues in photography theory has been the ‘reality effect’ of the 
photograph [17], that is to say that the photograph is not so much a created image 
as a recording, a document of what was in front of the lens at any given time. The 
reality effect is what lends photographs their ‘transparency’ [18], our tendency to 
look through photographs as material media to focus solely on what is depicted within 
the frame. The emphasis on transparency is what drives technological development, 
a quest to create a photograph which will be as real as being there, a pure and 
unadulterated ‘slice of time’. As technological development makes this more possible 
it is little wonder that photographers expand their thoughts into new territory and 
tend to explore the different forms of picture-making that the camera may afford. 

Elements of Francois Laruelle’s thesis [19] can be brought to bear on the deliberate 
error in this regard. Laruelle attempts to disentangle photography from its grounding 
in doubling or copying by pointing out the philosophical distinction between the 
object in the world and its photographed representation [19]. Perceiving the object and 
its photographed form as the same is a fallacy that over-prioritizes the world and the 
objects in it, without considering the distinct ontology presented by the photographic 
image. For Laruelle claiming the non-identity of the photograph releases it from the 
dualities of the ‘reality effect’ and permits it to be a related but non-identical depiction 
of objects in the world. Set free in this way the photograph is no longer the poor hand- 
maiden of reality but another presentation of reality which is distinct and unique to 
photography as a material practice [19, p. 55]. 

The potential of the photograph as a unique form of envisioning is suggested 
by Laruelle’s commentary, and speaks of the practical understanding of photogra- 
phy as a transformative art. While embedded in an examination of the photograph 
as a philosophical problem, Laruelle’s commentary expands photography from a 
simplified and primarily technological act toward an understanding of the central 
importance of the photographer/artist who, through the “spontaneous philosophy” 
of practice can reveal the “very essence of photography” [19, p. 85]. The photog- 
raphers exploring the photographic subversion reveal the ambiguity at the heart of 
photography by making images that run counter to the principle of representation. 

The creation of photographic subversions often centers upon an expansion of 
some kind: extended physical movement or elongated shutter times, which result in 
strange depictions of seemingly other-worldly phenomena—traces, trails, lines, and 
blurs which suggest but do not signify; the original scene lost behind a veil of color 
and movement (Fig. 9.6). These images present a divergent concept of photography 
which does not adhere to conventional understanding. They are often described as 
‘painterly’, a direct response to the evidence of gesture and time that are inscribed 
on the image. They are in many respects, a form of ‘non-photography’ divested of 
the need to stand in for things in the world, instead presenting another reality which 
is lost to our normal human vision. For those who seek a deliberate error there is no 
need to make a photograph per se—we have machines that can do that for us. For the 
artist the liberation lies in the expanded capacity of the camera as a tool for seeing 
the world in radically revised ways. 
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Fig. 9.6 Image copyright © 
Una Li and reproduced by 
permission. Source: Jn 
Pursuit of Error [8] 


The satisfaction of the accident, and of being presented with the will of the mate- 
rial (or technology), is an intrinsic part of an artist’s developmental method, their 
motivation to explore and progress. ‘Not-knowing’ drives practice forward [20]. 

In photography, the subversion contrives opportunities for not-knowing and to be 
surprised by unexpected outcomes. While a certain set of actions are put in motion 
it is not the case that the photographer can know how and to what extent they will 
affect the resulting image. The photographer may know that certain operations or 
actions are likely to produce one type of result, but she cannot predict the image 
outcome as when making a photograph in the conventional way. The subversion lies 
further toward the knowledge/control end of the continuum discussed at the start of 
this section, but it shares similarities with the accident in the gap that opens between 
intentions and outcome. This gap is made up of the present tense performance of 
photographing and the combination of photographer, camera, light, time, subject, 
and situation which coalesce in the resulting image. The error is an improvisation, 
always occurring in the immediate, unscripted moment and thus unknowable until 
that moment has passed. 
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For photographers who engage in the deliberate subversion of photographic pro- 
tocol the results of their transgression offer a similar level of satisfaction to encoun- 
tering an accident, and can perhaps offer new avenues to explore in practice. The 
possibility of a different type of photography is offered by the artists who tread this 
path. 


9.5 Tinkering with Technology 


The foregoing has shown how the error steps outside the conventions of photographic 
practice and reveals a more complex relationship between camera and photographer 
than that between an ‘operator’ and her ‘apparatus’. Photographers often have per- 
sonal relationships with their cameras, forming a kind of symbiosis between actants 
where one knows what a particular camera will do in a given situation. The pho- 
tographer who engages in “camera vision’ connects to the photographic seeing that 
is a required element of making good pictures and yet as we have seen this can be 
disrupted by an unexpected event or knowingly cast aside. These apparent ‘failures’ 
could be seen as end points, or dead ends from which one must reverse or retreat, 
or they could be seen as opportunities presenting new phenomena which stimulate 
the imagination. Invariably the contributors to IPE couch their errors in this latter 
context, as intriguing surprises which stimulate their practice or their perception 
about what a photograph is or could be. The etymological root of ‘error’ lies in the 
notion of ‘wandering’, suggesting a deviation which is both intended and imbedded 
in not-knowing as a principle [21, p. 21]. To wander is to be open to the possibilities 
that arise in that act of wandering, be they positive or negative and might also imply a 
return to the conventional route at a later point. This conception of error encapsulates 
how photographers use the error in practice to lay claim to previously unexplored 
territory. 

The territory occupied by the standards and procedures for ‘correct’ photography 
is vast and well developed. Since the very early inception of the medium, pho- 
tographic practice has been intrinsically tied to rules on the right way to use the 
technology in order to produce good photographs. Some guidelines were initially 
necessitated by the complex and awkward materials and processes entailed in early 
photography, but rapid development of photographic substrates meant that processes 
soon became easier, quicker, and cheaper. Rules are not made by technology itself, 
but by the culture that surrounds and uses it. In the case of photography, the empha- 
sis on progress and refinement exemplified through rapid technological development 
worked alongside a perceived need to regulate and define the limits of this emergent 
art form. Like all technologies, photography is deeply embedded in the social and 
political structures that helped formulate it as a popular and commercial entity and 
which continue to inform its development. In this network of influences, cultures of 
practice emerge. 

The most pertinent distinction in the practice of photography is between the 
informed and the casual photographer, a distinction evident since the early days 
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of the medium’s history. The informed amateur photographer was an early adopter 
of new technology, in some cases contributing to the development of the medium 
through their experiments. Informed photographers created Photographic Societies 
where they could discuss and disseminate treatises on the art and science of photog- 
raphy, organizing its philosophical principles, its areas of concern, and its goals. The 
casual photographer emerged as the medium became simplified and democratized 
by the new easier to use substrates and cameras, culminating in the Kodak, which 
offered them a means to enter the realms of photography without the necessary skills 
and knowledge to create their pictures other than a willingness to point and shoot. 
The first fully-fledged ‘movement’ in photography, Pictorialism, emerged as a reac- 
tion to these new ‘snapshooters’ and to the automation and loss of human skill that 
the new developments suggested for the medium [22]. 

Thus the technological development of photography, driven by democratization 
and commerce, contributed to a fracturing of perceived elements of skill and knowl- 
edge that were in the guardianship of a cognoscenti. Kodak’s invention forced the 
Pictorialists towards increasingly ‘unphotographic’ works—those that capitalized on 
the artist/photographer’s hand to destabilize the verisimilitude of the image, turning 
it instead into proto-painting [22]. The casual photographers, freed from concern 
about the quality of their images by the fast and easy methods available to them, took 
more chances with their photos; and errors, mistakes, and mishaps became part of 
the photographic landscape [23]. As Hand points out, this entry of the banal mistake 
into the culture of photography practice challenged “the dominant discursive and 
material legitimacy of photography, especially in terms of skill and expertise.” (24, 
p. 104]. 

Arguably, this distinction between informed/casual photographers is still recog- 
nizable today. The camera clubs of Britain are filled with those who pride themselves 
on their technical skills and expertise while casual photographers take selfies on their 
mobile phones. Photography’s intrinsic link with technology makes this distinction, 
and the value placed on technical know-how, continually pertinent today. 

The distinction between an informed and a casual photographer therefore lies in 
the amount of control they are able or willing to exert over the technology of pho- 
tographing, which includes the camera, recording medium, and image production. 
The informed photographer, through her expertise, exercises control over the tech- 
nology, mastering its principles to produce ‘correct’ photographs. The casual pho- 
tographer is a receiver of the technology and perhaps somewhat subservient to it in 
that its expertise appears to outstrip his own. With less emphasis on control needed 
or desired by the casual photographer, technological development for this market 
expands with handy, time-saving innovations which bring the ‘ideal’ photograph 
within easy reach. 

Thus, two paths of activity are suggested by cultures of practice in photography in 
which the level of control over the events of photographing favor one or other actant. 
As has already been demonstrated, the error appears to undermine this dichotomy and 
suggests a third way of working with photographic technology which is akin to Sherry 
Turkle’s concept of ‘soft mastery’ [25, p. 56]. This concept emerged from a study 
carried out by Turkle and Papert on student programming styles [26] which revealed 


154 T. Piper-Wright 


differences between those who took a logical, top-down approach to a programming 
problem, and those who preferred to work in an iterative, exploratory way with the 
computer. The distinction between ‘planners’ and ‘bricoleurs’ was expressed in a 
willingness, or otherwise, to engage with the computer as more than just a ‘tool’ but 
as a collaborator in the process of developing the program, exploring the nuances 
of its unique language and exploiting novel solutions to problems. Observation of 
different programming styles revealed that two approaches were at work; on the 
one hand there was a level of engagement with the process of programming as a 
problem which unfolded in time, and on the other, an approach which sought to pre- 
visualize all the aspects of the program before commencing work. This distinction in 
approach revealed a difference in the quality of the relationship between programmer 
and computer, which could be characterized for the bricoleur as a dialog and for the 
planner as a ‘monologue’ [26, p. 136]. The notion of conversation entails responding 
to the other party and perhaps adapting the flow of the conversation in accordance 
with those responses, while a monologue implies a silent audience. When errors 
occurred in the programming process, planners and bricoleurs differed markedly 
with how they dealt with this, either as failure or as opportunity: 


For planners, mistakes are missteps; for bricoleurs they are the essence of a navigation by 
mid-course correction. For planners a program is an instrument for premeditated control; 
bricoleurs have goals, but set out realize them in the spirit of a collaborative venture with 
the machine [26, p. 136]. 


Turkle’s concept of ‘soft mastery’ expresses a position of informed knowledge 
about the goal or the object to be realized, but an epistemologically different way of 
making the journey. Soft mastery relies on a form of negotiation with the materiality 
of the object in question, be it a computer or a camera, a programming language 
or abstract problem. Turkle adapts the concept of the bricoleur from Claude Levi- 
Strauss’s descriptions of knowledge creation in primitive societies, what he terms 
a “science of the concrete’ [26, p. 135]. To be concrete is to be present, tangible, 
handleable, not abstract and conceptual. This notion of the presence of objects as sets 
of things that can act and be acted upon is pertinent to a different way of understanding 
the camera, not as a tool to be mastered conceptually but as a thing to be worked 
with physically by testing it out and seeing what it will do. 

This approach is readily witnessed in photographers who experiment with the cam- 
era and suggests a level of playfulness and willingness to “see what happens’ when 
using the technology. This kind of ‘tinkering’ with the parameters of photographing 
can be seen in many of the contributions to IPE. It’s important to note that Turkle’s 
concept of ‘soft mastery’ does not set out to simply reverse the dichotomy between 
logic and practice, control and experimentation, but to expose the hierarchy that 
would assume that logic and control were the only or best ways of engaging with 
technology. The type of knowledge created through negotiation, through practice, 
reveals alternative epistemologies that run counter to the status quo [26]. 

A good example of a bricoleur approach to photography can be found in the work 
of Julia Margaret Cameron. Cameron was an early pioneer of photography as an 
art form, creating striking portraits of friends and family in scenarios influenced by 
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literature and painting. A particular feature of Cameron’s photography is a deliberate 
softness of focus. In her extensive writings she notes how this particular feature of 
her work arose in practice, and led her to adopt it as a signature style: 


I believe...that my first successes in my out-of-focus pictures were a fluke. That is to say, 
that when focusing and coming to something which, to my eye, was very beautiful, I stopped 
there instead of screwing on the lens to the more definite focus which all other photographers 
insist upon [27, p. 77]. 


It is pertinent that while having complete command of the camera as a piece of 
technology, Cameron instead chose to pursue a deliberate ‘mistake’ or subversion 
of the accepted protocols for photographing in order to compose an image which 
was closer to her intentions. Cameron’s informed but embodied interaction with the 
camera was deemed ‘sloppy’ and incompetent by those who had set themselves 
as gatekeepers for the craft, a regulatory status quo which set the rules by which 
photographs were to be made [28]. The fact that Cameron refers to them as her ‘out 
of focus pictures’ is perhaps a significant internalization of this criticism in a nascent 
world of cultures of practice which were being formed and entrenched. 

What Cameron’s early flouting of the ‘rules’ of photography suggests is a connec- 
tion between photographer and camera in which the photographer becomes aware 
of the frailties and limitations of the technology, and the creative potential that these 
peripheries afford. This awareness stems from a willingness to get inside the camera 
in some way, to envisage its “camera seeing’ as a unique form of agency which has 
a contribution to make to the final image. In their study Turkle and Papert observe 
this type of investment into inanimate objects as a form of “reasoning from within” 
[26, p. 144] which is embodied rather than abstract. Reasoning from within requires 
a proximal relationship with the objects that we use, rather than a distance which 
views them as purely functional tools. To acknowledge the camera’s presence in 
the making of a photograph is to acknowledge it as a collaborator and changes our 
relationship with it. 

Entry to this new way of thinking about technology might be as simple as the 
first time we encounter something that has gone wrong. Bill Brown observes that we 
rarely notice the objects that surround us when they are working correctly. In this state 
they become invisible; we almost see past them, or through them. However, when 
something goes awry we are forced to reckon with the object as a different order 
of ‘thing’, an agent that has impacted on us in an unforeseen way. The breakdown 
or interruption in the flow of expected behavior changes our relationship with the 
object and redefines it [29]. 

Error, mistakes, tinkering, and subversion are therefore vital and important aspects 
of photography as a creative practice. They create gaps in the armor of accepted 
practice, chinks of light where other possibilities for photography can emerge and 
be explored. The epistemological pluralism espoused by Turkle and Papert [26] 
exposes a new way of relating to photographic technology; as an embodied, proxi- 
mal relationship between two agents, two actors who negotiate in the creation of a 
photograph. 
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9.6 Conclusion: Being in Two Minds 


The foregoing has suggested some alternative ways to consider the relationship 
between camera and photographer, and how cultures of practice emerge which rein- 
force oppositional hierarchies between users and their tools. The power of the acci- 
dent or the subversion to undermine some of the dominant metaphors that adhere to 
photography—facsimile, accuracy, perfection—exposes a counter culture of photo- 
graphic practice which has been explored and exploited by practitioners since the 
early days of the medium. 

To adopt the embodied and proximal methodology of the bricoleur is to be open 
to a sense of discovery and unfolding in the creative process, a form of confident 
‘not-knowing’ from which to begin. 

Vilem Flusser observes that “The act of photography is that of ‘phenomenolog- 
ical doubt’.” [2 p. 38]. For Flusser doubt in photography is expressed through the 
potential multiple viewpoints enabled by the camera and the aspects of the photo- 
graphic process which are invisible to us. Doubt is woven into the elements of the 
photographic event in which multiple actants converge at a moment in time and in 
the hiatus of control and decision-making that is characterized by the accident or the 
subversion. 

The etymology of doubt in ‘doubling’ in the sense of having to choose between two 
things [30] offers another way of interpreting the not-knowing that we experience in 
the event of photographing. To doubt is to be in two minds or to assume two positions. 
The error exposes this doubling as it occurs in practice, as a dance between two 
epistemological positions: to know and not know, to be in control and to relinquish 
control, and as a dance between two actors: camera and photographer. To be in 
two minds is to enter a position which permits a relational flow between machine 
vision and the embodied photographer, between presence and program, in which the 
merging of both exceeds the power of either. 


References 


1. Deleuze, G.: Bergsonism. Translated by H. Tomlinson and B. Habberjam. Zone, New York 
(1988) 

2. Flusser, V.: Towards a Philosophy of Photography. Translated by A. Mathews. Reaktion, 
London (2000) 

3. Parker Willis, N., Porter, T.O. (eds.): The Pencil of Nature. A New Discovery. The Corsair. 1 
No. 5 (Saturday April 13): 70-72. Reprinted in Photographic Theory: An Historical Anthology. 
In: Hershberger, A. (ed.) Wiley-Blackwell, Oxford, pp. 44-47 (2014) 

4. Da Vinci, L. C.: The Function of the Eye. As Explained by the Camera Obscura in Photographic 
Theory: An Historical Anthology. In: Hershberger, A. (ed.) Wiley-Blackwell, Oxford, pp. 17- 
18 (2014) 

5. Petchesky, R.P.: Fetal Images: The power of visual culture in the politics of reproduction. Fem. 
Stud. 13(2), 263-292 (Summer, 1987) 

6. Pritchard, M.: A History of Photography in 50 Cameras. Bloomsbury, London (2016) 


9 Between Presence and Program: The Photographic Error ... 157 


T 


27. 


28. 
29. 
30. 


Bate, D.: The digital condition of photography: cameras, computers and display. In: Lister, 
M. (ed.) The Photographic Image in Digital Culture, 2nd edn. Routledge, London, pp. 77—94 
(2013) 

Piper-Wright, T.: In Pursuit of Error [Online]. www.inpursuitoferror.co.uk 

Frow, J.: A pebble, a camera, a man who turns into a telegraph pole. Crit. Ing. 28(1), 270-285 
(Autumn 2001) 

Feenberg, A.: A critical theory of technology. In: Felt, U., Fouché, Clark, R., Miller, A., 
Smith-Doerr, L. (eds) The Handbook of Science and Technology Studies, 4th edn. MIT Press, 
Cambridge Mass, Cambridge, MA, pp. 635-663 (2017) 


. Latour, B.: On actor-network theory: a few clarifications. Soziale Welt Jahrg H 4, 369-381 


(1996) 


. Borda, S.G.: Locating the camera in a digital time. Photographies 11(2-3), 177—192 (2018) 
. Bennett, J.: Vibrant Matter, A Political Ecology of Things. Duke University Press, Durham, 


NC (2010) 
Chong, D.: Tokyo 1955-1970 A New Avant Garde. The Museum of Modern Art, New York 
(2012) 


. Meatyard, R.E.: Learning to see ‘No-Focus’ (2011) [Online]. https://www.americansuburbx. 


com/2013/01/ralph-eugene-meatyard-reframing-photography-theory-and-practice-excerpt- 
2011.html 


. Barth, U.: Artist’s website [Online]. http://utabarth.net/work/untitled-1998-99/ 
. Barthes, R.: In: Barthes Reader, A., Sontag, S. (ed.) The Photographic Message [1961]. Hill 


and Wang, New York, pp. 194-210 (1982) 


. Walton, K.: Transparent pictures: on the nature of photographic realism. Crit. Ing. 11(2), 246- 


277 (1984) 


. Laruelle, F.: The Concept of Non-Photography. Translated by R. Mackay. Sequence Press, New 


York (2011) 


. Barthelme, D.: In: Herzinger, K. (ed) Not Knowing in Not Knowing: The Essays and Interviews 


of Donald Barthelme. Random House, New York (1997) 


. Nunes, M. (ed): Error: Glitch, Noise and Jam in New Media Cultures. Continuum Books, 


London (2011) 


. Warner Marien, M.: Photography: A Cultural History, 4th edn. Laurence King, London (2014) 
. Hirsch, R.: Seizing the Light: A History of Photography. McGraw-Hill, Toronto (2000) 


Hand, M.: Ubiquitous Photography. Polity Press, Cambridge (2012) 


. Turkle, S.: Life on the Screen: Identity in the Age of the Internet. Simon & Schuster, New York 


(1995) 


. Turkle, S., Papert, S.: Epistemological pluralism: styles and voices within the computer culture. 


Signs 16(1), 128-157 (1990) 

Cameron, J.M.: Annals of My Glass House. First Published in Photo Beacon (Chicago) 2 
(1890), pp. 157-160. In: Hershberger, A. (ed.) Photographic Theory: An Historical Anthology. 
Wiley-Blackwell, Oxford, pp. 76-79 (Reprinted) (2014) 

Weaver, M.: Julia Margaret Cameron 1815-1879. Little Brown Company, Boston, MA (1984) 
Brown, B.: Thing theory. Crit. Inq. 28(1), 1-22 (Autumn, 2001) 

Klein, E.: A Comprehensive Etymological Dictionary of the English Language. Elsevier, 
London (1969) 


158 T. Piper-Wright 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Chapter 10 A) 
Signs of Surveillance get 


Daniel Buzzo 


Abstract This chapter discusses the challenges of working with complex digital 
information, visual images, meta-data, and temporal and geo-spatial information, as 
raw materials for artistic expression. In the chapter the reader will have an insight into 
the thought process and working practice, artistic and technical, of forming informa- 
tion into a recognizable body of work and processing both the content and the form of 
the work into an emotional, reflective experience. It not only discusses the challenges 
of working with external, commercial software frameworks and web services and 
the challenge of dealing with the brittleness of digital things but also the ‘semantic 
affordance’ offered by computer systems in working at a conceptual level with art 
materials. The ‘Signs of Surveillance’ project originated as a photographic observa- 
tion and collation activity in early 2015 and has grown to a complex digital web and 
installation live visualization project. Since that time thousands of photographs of 
signs of surveillance and the warning signs, indicating an area or activity is being 
monitored by video-camera, have been captured in more than 15 countries, including 
Belgium, Canada, China, Denmark, France, Germany, Greece, Italy, Japan, Korea, 
Luxembourg, Netherlands, Portugal, Spain, Sweden and the UK. This chapter dis- 
cusses the development of one aspect of this multi-part work, dealing directly with 
managing and manipulating a large body of digital data and working with complex 
visualization systems and online and offline digital distribution technology. 


Keywords Digital art - Media art - Installation - Semantic affordance - Digital 
brittleness - Surveillance society - Derive + Geo-location + Web technologies + 
Visualization 


10.1 Introduction 


The pervasive nature of surveillance, as well as the normalization of the signs of it— 
both physical and metaphorical, is arresting when it is laid out how clearly intertwined 
this silent over-watch of endless surveillance cameras has become with our everyday 
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lives. Across Europe and the wider globe, surveillance of the body public, of civic 
space, of every interaction in everyday society appears at some kind of saturation 
point.! In 2015, the then UK Information Commissioner, Tony Porter, himself a 
former counter-terrorism officer said: 


The lack of public awareness about the nature of surveillance troubles me 


and as Matthew Thomas wrote in the Guardian newspaper, quoting Porter: 


[it] risks changing the “psyche of the community” by reducing individuals to trackable 
numbers in a database 


This is despite strong evidence that the publicly stated goals of civic safety and 
crime prevention are not being fulfilled [25]. The 2005 UK Government Home Office 
Research, Development and Statistics Directorate concludes in its official report, 
‘Assessing the Impact of CCTV’ [14] that: 


the CCTV schemes that have been assessed had little overall effect on crime levels [...] 
CCTV is an ineffective tool if the aim is to reduce overall crime rates and make people feel 
safer. The CCTV systems installed in 14 areas mostly failed to reduce crime (with a single 
exception), mostly failed to allay public fear of crime (with three exceptions) and the vast 
majority of specific aims set for the various CCTV schemes were not achieved. 


In 2004, Richard Thomas the then UK Government Information Commissioner 
warned [2]: 
My anxiety is that we don’t sleepwalk into a surveillance society where much more infor- 


mation is collected about people, accessible to far more people shared across many more 
boundaries, than British society would feel comfortable with 


Making a comparison to the collection of vast quantities of information on indi- 
vidual of Franco’s fascist state and the eastern European communist regimes, he 
added in response to the question of whether there was a risk of Britain following 
this route, 


I think there is a danger. I don’t think people have woken up to what lies behind this. It 
enables the government ... to build up quite a comprehensive picture about many of your 
activities. 


Itis against this confusing and disturbing backdrop that this project asks a question 
of the surveillance colonization of the physical environment around us by building, 
visualizing and exploring an international database of signs of surveillance. These 
seemingly ubiquitous information/warning displays come in all shapes, sizes and 
designs, while often appearing helpful or informative is actually (in the UK at least) 
part of the legal obligation, largely unknown by the public and ignored by operators, 


'Tn its 2013 report, The Picture Is Not Clear: How Many CCTV Surveillance Cameras Are There 
in the UK? [3] the British Security Industry Association (BSIA) suggested that camera numbers in 
the private sector could outnumber those used by public bodies by as much as 70 to 1. The BSIA 
survey covered the whole of the UK, not just London, and its maximum estimate suggested there 
was a CCTV camera for every 11 people in the country though it said the most likely figure was 
closer to one for every 14 people. 
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on any operator or installer of a CCTV surveillance camera. The Information Com- 
missioners Office (ICO), the independent UK authority, explains in the guidance 
‘CCTV filming carried out by others: What can I expect?’ [23] 


The CCTV operator must let people know they are using CCTV. Signs are the most usual 
way of doing this. The signs must be clearly visible and readable, and should include the 
details of the organization operating the system if not obvious. (emphasis added) 


CCTV should only be used in exceptional circumstances in areas where you normally expect 
privacy—such as in changing rooms or toilets, and should only be used to deal with very 
serious concerns. The operator should make extra effort to ensure that you are aware that 
cameras are in use. 


As can be seen from the corpus of collected images in this project, this point 
that CCTV warning signs ‘should include the details of the organization operating 
the system’ is one that is notable when operator information is included rather than 
when it is absent, such is the apparently casual attitude to the use of CCTV and any 
adherence to privacy or regulatory policy. 

This growth of surveillance and reactions to it, rather than being a recent phe- 
nomenon, has been going on for nearly three decades. Several artists and projects 
have worked in this area; notable amongst them are recent projects such as ‘Art 
and Surveillance’ lead by Susan Cahill in Canada that collates artistic projects on 
surveillance [8] and older ones such as http://www.spotthecam.nl created in early 
2000s by Maurice Wessling and Bits of Freedom? in conjunction with the Waag 
Society in Amsterdam, Netherlands? and the ‘Surveillance Camera Theatre’* from 
the Surveillance Camera Players [24], an informal activist theatre troupe formed 
in 1996 in New York City between Bill Brown and Michael Carter. Carter, in his 
1995 manifesto “The Guerrilla Programming of Video Surveillance Equipment’ [9], 
proclaims: 


It is important to remind oneself of the relationship between the eye of the media and that 
of the corporate police state — for they are both the guardian of the commodity, however 
nebulous and ephemeral that commodity may become. As a tactic designed to point out 
the paradox of a system that turns the lens on a public that has been taught to place more 
importance on images recorded by cameras than images seen by their own eyes, we propose 
Guerilla Programming of Video Surveillance Equipment (emphasis added). 


As a photographic observational project, the existing corpus of images of ‘signs 
of surveillance’ comprises over 2000 digital photographs taken across the globe 
between 2015 and 2019. Each image contains date, time and geo-location meta-data 
allowing geographic mapping and search and retrieval in a variety of ways. The 
whole project has several elements: 


e A compendium of the designs, locations and forms of the myriad of sign-types in 
the collection (see Fig. 10.1) 
e A traditional digital photographic print exhibition 


*https://www.bitsoffreedom.nl. 
Shttps://waag.org. 
4http://www.notbored.org/scp-how-to.html. 
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e An interactive 3D geo-map-based visualization 

e Anonline web-based searchable catalog that asks for public contribution to extend 
and complete the European map of Signs of Surveillance 

e Training a machine learning system on the corpus of signs to enable automatic 
sign recognition. 


This chapter deals with the development of element number three, the develop- 
ment of an interactive, dynamic 3D geo-map-based visualization of the photographic 
collection. 


10.2 Process of Gathering Raw Images and Meta-Data 


The process of gathering raw images for the collection of photographs of signs of 
surveillance began as a non-verbal act of noticing, in noticing the apparent over- 
abundance of these signs in public spaces. After noticing came the wordless desire 
to evidence that noticing, to collate and compare, to contrast and reflect. to investigate 
if the feelings of noticing were borne out by actual evidence—in this instance the 
evidence being the collation of photographs recording the signs. This process of 
observation and recording as basic research, and raw material, is common to many 
artistic practices, and is a recurring theme I use to undertake non-verbal investigation 
of items of importance I see in the world around me. In some ways I use the camera 
as one would use a microphone to take field recordings for later study in the studio. 
In this work the gathering of images made the proliferation of these signs in public 
spaces abundantly clear (see Fig. 10.1); partly as the act of looking for them became 
habit, partly as my subconscious eye became trained to see these small icons, images 
and visual devices and the kind of places they would appear high up on walls of 
large buildings, low down on shop doorways, in the entrances and lobbies of offices 
and houses. The act of looking also revealed the signs in unexpected places, among 
piles of pastries in display cabinets, buried in hedges and flower beds in residential 
gardens, on the sides of cars, buses, trams and trains, and on vending machines and 
garden sheds. 


10.2.1 Building Semantic Affordance with Artistic Raw 
Materials 


In the process of gathering the photographs, extending the corpus of images from 
country to country as I traveled while engaged in work on other projects, a feel for 
the body of work began to emerge. The feeling of emergent, artistic, comprehension 
is the one that I recognize when undertaking works of almost any sort, from extended 
drawing studies of scenes (see Fig. 10.2) or objects to wrestling with the philosophical 
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Fig. 10.2 Example of extensive use of drawing for exploration and visualization, here showing 
early explorations of using image meta-data and tagging to map-tile backgrounds 


aspects of how best to structure a system when incorporating dense computer code 
(such as the Volca project [6, 7] or the Time Machine [4, 5]). 

This understanding is the developing of a cognitive, sensory and implicitly non- 
verbal relationship with the materials and ideas or objects under scrutiny. It is a 
feeling of what I call ‘semantic affordance’, in that one feels an innate ability to 
cognitively manipulate the essential conceptual elements embodied by whatever is 
the subject of scrutiny. This affordance gives a direct fluidity and competence that 
allows further examination and contemplation of the materials at hand at a concep- 
tual and ontological level. This affordance or agency to manipulate the conceptual 
elements embodied by the material under study could be seen as the central activity 
in my artistic practice. It is through this level of deep non-verbal understanding of 
the manifold semantic concepts seen held within objects and materials that inform 
the aesthetic choices that are made when re-formulating materials in the prepara- 
tion and creation of artistic outputs. As the artistic outputs are imagined, sketched, 
produced as maquettes and prototypes, a re-connection to the original non-verbal 
enquiry is sought. The process is a search to close the loop from initial observation, 
non-verbal comprehension, manipulation and synthesis of conceptual structures to a 
final embodiment of concepts as an intervention into the physical world. The answer- 
ing of the initial question and the final scratching is a mental itch. In this sense the 
works are predominantly conceptually driven and operate at a physical level in the 
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conceptual rather than technical or decorative world. As Shanken [21] raises in ‘Art 
in the Information Age: Technology and Conceptual Art’, although the divisions 
between conceptual art and technological art have generally been seen as in con- 
flict, he explores the view that while the history of technological art has been seen 
as separated from that of conceptual art both were rooted in experimentation and 
re-interpretation and proposes a re-thinking of their status as separate and divorced 
from each other. He quotes Sol Lewitt’s 1967 essay ‘Paragraphs on Conceptual Art’ 
[16] where Lewitt describes conceptual art as a quasi-mechanical process: 


In conceptual art the idea of concept is the most important aspect of the work . . . [t]he idea 
becomes a machine that makes the art. 


Itis this very idea that drives my quest for these conceptual affordances in the work, 
which allows mental structures to be created from which the art creates itself. The 
arena of the technical in part is a desire to work unbounded by both convention and 
the patina of the physical that seem part and parcel of other mediums of expression. 
The ‘active’ nature of technology allows for embedding, encoding and subsequently 
animating these ‘conceptual machines to make art’ into the systems underlying the 
physical surface of the work. 


10.2.2 Gathering Images and the Dérive 


In this project the initial activity was a process of subconscious harvesting, con- 
ducted as an ongoing photographic dérive, a drift, though the world we inhabit with 
a watch for these signs of surveillance that had now become artistic items of interest. 
Guy Debord, French philosopher and author of “The Society of the Spectacle’ often 
associated with Situationist International,’ describes the dérive as: 


a mode of experimental behavior linked to the conditions of urban society: a technique of 
rapid passage through varied ambiances [13] 


Over the period beginning with the first photograph of the series, taking in Lisbon 
in June 2015, and lasting through 15 countries and more than four years over 2,000 
images of signs of surveillance were recorded in dérive fashion. 


10.3 Managing and Manipulating Large Digital Image 
Corpuses as Artistic Raw Material 


Dealing with materials gathered using digital tools as the subject of artistic study 
has a number of opportunities and also challenges. Some of these are common to 
the practice of much artistic research and some unique to digital materials. Digital 


Shttps://www.tate.org.uk/art/art-terms/s/situationist-international. 
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materials, while fluid, flexible, malleable, and almost instantaneously duplicable and 
transferable, are also at their core, inherently brittle. Their flexible, immaterial, nature 
and their ability to be versioned, trans-coded, summarized and edited means that 
any notion of an original is fleeting and slippery. Attempting to trace the origin, and 
original of a digital document, is like grasping at a ribbon blowing in the wind. Partly 
this is the nature of sampling the world with digital, that is, numerical processes. It 
is important to remember that the sampling process is fundamentally divorced from 
techniques of the analog recording realm.° The connection with the place and instant 
of creation is fleeting, and beyond the memory of the place and time that a button 
was pressed or a menu item clicked the digital object has no patina or evidence of 
its place or moment of creation. 


10.3.1 Meta-Data and Artistic Data Management 


All digital artifacts have meta-data, that is, there is data about the data. There is 
information about the information contained inside any digital document, be it sound 
file, image, text, video or software code. What is unusual is that in the digital realm 
this meta-data is built of the same substance as the mate‘riale it describes. In the 
instance of this project each image was recorded with meta-data describing the image. 
There are basics facts associated with any digital files, for example, creation date, 
modification date, byte size of the file, name, filetype and the current location of a 
document in a specific computer system. There also, commonly, exists within image 
files additional data using the EXIF (EXchangeable Image File format) standard.’ 
The EXIF data standard was originally introduced by The Japan Electronic Industries 
Development Association (JEIDA) in 1998 and is widely adopted as a standard to 
store extended information about digital photographs. This meta-data commonly 
includes information about the make and model of the apparatus or camera used 
for the original recording and extends to information about lenses, flash, aperture 
and shutter speed of the recorded image. The meta-data also includes contextual or 
circumstantial data including the date/time of the image capture and increasingly 
common with images taken with smart phones and the latest generation of hand-held 
cameras, the geographic meta-data, expressed as latitude, longitude and altitude. This 
EXIF data and specifically the associated geo-location data became of particular 
interest in the course of this project and is processed and manipulated heavily in the 
final artworks produced as part of the work (see Figs. 10.3 and 10.4). 


Sampling puts a mesh to the world and encodes numerically, explicitly sampling reality at positions 
on a pre-defined grid—the only parameters of this crude and bombastic approximation are the 
frequency and resolution of the grid and the depth of the data captured or sampled at each point. 
All information, texture and nuance between the sampling points of the grid are simply ignored or 
disposed of. 


Thttps://www.exif.org. 
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Fig. 10.3 Early map visualization using C++ and openFrameworks with spline paths generated 
following chronological order or recording of individual images. Here showing walks through 
Kyoto, Japan over a three day period in 2018 


Fig. 10.4 Early map visualization using C++ and openFrameworks with spline paths generated 
following chronological order or recording of individual images. This version utilizes only geo- 
location and temporal data from each image 
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10.4 Developing Artworks Combining Hand-Coded 
and Industrial Digital Tools 


Developing artworks utilizing technology, whether that be industrially mass- 
produced physical materials such as paint, canvas, glass-fiber, electronics or soft- 
ware involves a level of compromise and negotiation with the materials an artist 
chooses to engage with. Being restricted to colors manufactured by a supplier, being 
constrained by the retail costs of a device or being challenged by the features avail- 
able in a software API are all within the same canon of challenges faced by artists 
working in many contemporary contexts. As an artist regularly working in depth 
with technology, and software specifically, this challenge of adapting, working with, 
fighting with and customizing industrial products is a regular occurrence. Part of the 
aesthetic of much technology orientated art is this engagement with, and subversion 
of, industrial products, situating them in new contexts and utilizing them in new and 
novel ways. Implicitly tied to this aspirational goal of both mastery and subversion 
comes a challenge within the practice of making art with technology, one that is 
endemic to all art but is amplified exponentially when engaging with software. This 
challenge is the negotiation, both in production and distribution of works, with the 
inherent brittleness of digital things. 


10.4.1 On the Brittleness of Digital Things 


As all physical things decay and change under what appears to be the effect of time,* 
so do digital ones, and this decay is seemingly at a rate inversely proportional to 
the power of digital material. Just as a digital image can be spread around the world 
and seen and manipulated by a billion people simultaneously it can also collapse, 
cease and disappear with equal speed. A server outage, a power failure, a change 
in the software application programming interface (APT)? or even the simple act of 


8One could begin a lengthy discussion in the nature of the perception of time, correlation and 
causality. Suffice to say that in most practical and functional aspects it would appear that as Stappers 
and Giarcardi point out in their work on designing for temporal context [22]: 


The most important thing designers should know is that there is no such single thing as time. 


As Carl Sagan says “[Time] is profoundly resistant to simple definition.” Discussing the nature 
of time, Sagan, in an interview during the making of NOVAs TV program ‘Time Travel’, goes on 
to note [20]. 


Ever since St. Augustine, people have wrestled with this, and there are all sorts of things it 
isn’t. It isn’t a flow of something, because what does it flow past? We use time to measure 
flow. How could we use time to measure time?. 


°In computer programming, an application programming interface (API) is a set of subroutine 
definitions, communication protocols, and tools for building software. In general terms, it is a set 
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applying a seemingly unrelated operating system update can bring digital artworks, 
not to their knees, but to a state of ceasing to be. Though digital files and lines of 
software code may still be intact, software art, like video art before it, is performative 
and temporal; it exists at the moment of viewing; it needs to be run, to execute, to 
function and to fulfill its promise as art. This base need of instantaneous creation, to 
generate itself at the moment of viewing by an observer, is the challenge, excitement 
and weakness at the heart of digital artworks. Bit Rot, [10, 15] or occasionally Link 
Rot,!° is described by Coupland in his 2016 book of the same name [12] as: 


the way digital files of any sort spontaneously (and quickly) decompose 


The most important thing designers should know is that there is no such single 
thing as time. 

As Carl Sagan says “[Time] is profoundly resistant to simple definition.” Dis- 
cussing the nature of time, Sagan, in an interview during the making of NOVAs TV 
program ‘Time Travel’, goes on to note [20]: 


‘Ever since St. Augustine, people have wrestled with this, and there are all sorts of things it 
isn’t. It isn’t a flow of something, because what does it flow past? We use time to measure 
flow. How could we use time to measure time?’ 


and is ‘a slang term’ for hypertext links that are broken. Link rot is created when 
a web page is moved, taken down or re-organized. Clicking on a rotten link usually 
results in a 404 error, which includes a message that the page cannot be found. This 
‘digital decay’ affects all computational systems and is in equal parts the ventral 
thrill and motivation and also an ever-present challenge in digital art making. As the 
internet pioneer Vinton Cerf observes [10]: 


There is something ultimately satisfying about keeping information in digital form... But 
this blissful outlook may not comport with the reality of digital information preservation and 
interpretation. 


In the context of this artwork I am writing about—-sat in seat 6c on a flight to 
Lisbon, tapping on a tablet computer (and now weeks later re-editing on a subway 
train under Hong Kong harbor)—hoping that what I write here will synchronize 
flawlessly with the master version for this document so I can continue on my laptop 
at home, or the desktop computer in my office or the myriad of other computers I 
have in my two studios—the challenge of working with industrial tools in the making 
of signs of surveillance has been great. Both in the practicality of the making of the 


of clearly defined methods of communication among various components. A good API makes it 
easier to develop a computer program by providing all the building blocks, which are then put 
together by the programmer. An API may be for a web-based system, operating system, database 
system, computer hardware or software library. An API specification can take many forms, but 
often includes specifications for routines, data structures, object classes, variables or remote calls. 
POSIX, Windows API and ASPI are examples of different forms of APIs. Documentation for 
the API is usually provided to facilitate usage and implementation. (https://en.wikipedia.org/wiki/ 
Applicationprogramminginterface). 

!Ohttps://www.techopedia.com/definition/204 14/link-rot. 
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work, designing, programming and forming it, and also in the conceptualization of it, 
the philosophical and aesthetic decisions underpin it structurally. This arena is the ill- 
defined and rarely talked about area where semantics and ontology meet engineering, 
where philosophy becomes embodied in systems and structures and ways of doing 
things. Just as William Gibson says [1] with incisive clarity: 


We can’t see our culture very well, because we see with it 


Perhaps this sentiment is distilled to its essence in the Japanese anecdote that ‘to 
a hammer, everything looks like a nail’. 


10.4.2 Aesthetics, Frameworks, APIs and Ontologies 


When looking through the gathered body of work, mentally sitting among the thou- 
sands of photographs from around the world, and reflecting on the journey of col- 
lection and understanding, a non-verbal set of connections start to be made, there 
begins to form an ontological framework of what is and what is not, what does 
go and what does not when considering, comparing, organizing and arranging the 
material. Alongside the challenges in managing digital collections, of cataloguing, 
versioning, trans-coding, naming, renaming and duplicating and archiving, there 
is the selection of applicable tools to manipulate these non-physical collections of 
things-that-may-be. 

In the development of the work the public exposure of the meta-data inside each 
image became increasingly important in relating an individual item in the collection 
to the whole. Beginning with a process of drawing, commonly with ink on paper, 
ideas of collating, relating and visualizing this relationship, between individual ele- 
ment and the whole, took shape. Sketching ideas in code, using the C++ toolkit, 
openFrameworks,!! a sense of the dynamic in the material came to the fore. Initially 
the project was an observation and dérive exercise and at a certain stage it trans- 
formed into a discussion with the gathered material. As is of often the case in life 
drawing, one starts with mark making in reference to the subject being observed but 
at some point crosses into mark making in reference to the body of marks one has 
made on the paper or canvas. Adding to or subtracting from a growing object that 
begins to exist independently of that which is originally observed. This is the point 
where a piece of work develops its own life or emergent identity, one that is divorced 
from being a facsimile of the original scene. 

After working with initial crude mappings of the geo-located images into 3D 
space using geometric primitives and texture mapping of a variety of representa- 
tions of the earth’s surface a move to a more complex visualization was needed. To 
access high-resolution maps and images of the surface of the whole earth necessi- 
tated engaging with external data sources, initially continuing working writing code 


'l<QpenFrameworks is an open source C++ toolkit designed to assist the creative process by 
providing a simple and intuitive framework for experimentation’ (https://openframeworks.cc/ 
about/). 
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in C++, the open-source online OpenStreetMap (OSM).!* Geographical informa- 
tion system (GIS) database was accessed through its web-based API.!* While this 
brought huge flexibility and access to a wealth of visual and meta-data about the 
earth it added a subtle but pivotal disjuncture in the work. Even though the code was 
still handwritten C++, compiled to run locally as an executable binary application on 
a specific machine, the work now had a brittle chain of failure built into it. The work 
now became reliant on an internet connection, reliant upon a connection of sufficient 
bandwidth to the outside world and at the other end of a chain of routers and con- 
nections. It was reliant upon the servers at OSM being available, and that the access 
restrictions and protocols of their API, and the data in their underlying database 
were available and had not been altered, patched, upgraded or had their license of 
use changed. What also began to happen is that the underlying ontology, the causal 
philosophy of the interrelation between elements, images, ideas and concepts that 
was embodied in the architecture of the software I was writing changed. The atti- 
tude and values in the project, the structure, classifications and ontology that was 
emerging, was codified in its most concrete form in the classes, methods, functions, 
loops, arrays, iterators and variables that scaffolded and guided the work to become 
at the moment of performance. By engaging with this external API, and the underly- 
ing philosophy, attitude, ontology and expectation encoded within it, it there began 
and evolving and protracted negotiation on how to navigate the external conceptual 
space without becoming overwhelmed hegemonistically!* and becoming merely an 
extension of it. Illustrating what it could do, rather than a master of it demonstrating 
what I could do with it, sculpting it to my will and the will of the evolving work. This 
conundrum is somewhere between the Sapir-Worf hypothesis [17] on how language 
shapes thought and conceptions of one’s reality and the ominous words of Friedrich 
Nietzsche when he said [18]: 


Whoever fights monsters should see to it that in the process he does not become a monster. 
And if you gaze long enough into an abyss, the abyss will gaze back into you. 


This negotiation for control became a battle, metaphorically, for the ground under 
the project, and this external empire of protocols and ontologies and the emerging 


'2http://www.openstreetmap.org. 


'3 Utilizing the openFrameworks add-on ofxMaps (https://github.com/bakercp/ofxMaps). ofxMaps 
creates an interface to OSM in the ‘SlippyMap’ style (https://wiki.openstreetmap.org/wiki/ 
SlippyMap). It is written and maintained by Christopher Baker. 


'4The Encyclopedia Britannica describes Hegemony as the dominance of one group over another, 
often supported by legitimating norms and ideas. The term hegemony is today often used as short- 
hand to describe the relatively dominant position of a particular set of ideas and their associated 
tendency to become commonsensical and intuitive, thereby inhibiting the dissemination or even 
the articulation of alternative ideas. The associated term hegemon is used to identify the actor, 
group, class or state that exercises hegemonic power or that is responsible for the dissemination of 
hegemonic ideas. (https://www.britannica.com/topic/hegemony). 
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artwork—using C++ but communicating into a web-based world, using http trans- 
port!> layers—began to shape, not only what was being said but a shift linguistically 
into the world of web technologies. That is not to say that the possibility to stay 
in the dialect I was writing in was not available, it was entirely, but the challenge 
would be to re-write for myself the additional tools, utilities, libraries and processes 
I was beginning to rely on to achieve the realization of my increasingly complex and 
specific pen and ink drawings of the project. Just as a decision to make your own 
paint becomes a significant part of the work so does the decision to make one’s own 
digital tools. 

As Richardson discussed in his 1998 article ‘New Media, New Craft’ [19] 
reflecting on what the status of new digital tools may be in an art context: 


Using programming to create a piece of art or design requires an abstraction of thinking, 
translating the idea of the final visual form into a structure required to create the work. A 
leap of understanding is required to translate a creative idea into a piece of code. Creative 
ideas and solutions are thus abstracted into structures and objects that are natural to the 
computer material. Such an understanding of material subsequently creates work that echoes 
the invisible structure of the code 


Richardson goes on to add: 


The use of programming, as a way of manipulating and understanding the new material, 
also represents the means by which artists and designers are able to get closest to the virtual 
material. When they use programming to create a visual work, the underlying process and 
structure become of fundamental importance. The invisible structure of the work becomes as 
significant as the final outcome. The work experienced by the viewer is a visual translation of 
the underlying framework and mental engagement of the artist or designer with the material. 


This challenge, between using available tools, and traveling further, faster and 
making everything from scratch for oneself, perhaps risking re-inventing the wheel 
having taken years to learn how to best make a wheel, is common to many artistic 
endeavors but never more so than in self-coding digital artworks. Richardson contin- 
ues pointing out the difference that hand-crafted digital tools make in the conceptual 
understanding of any artwork: 


In direct contrast to the post-modern point-and-click, cut-and-paste approach to creating 
a piece of creative work, the use of programming requires that a framework is firmly 
established, around which the work is built. 


Having made strategic choices and seen where the project wanted to go, and 
how it could be formed in a way that allowed it to talk for itself, investigations 
of middleware frameworks and software libraries began in earnest. Moving parts 
of the codebase into java script and using the browser as interpreter and platform 
wheel, rather than the byte code produced by CLANG or the GCC C++ compiler 


'Shttp, or hypertext transport protocol, is a computer protocol that is used to allow clients (commonly 
web browsers running on users computers to request and subsequently receive text, image, audio 
and video data from web servers). 
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allowed easy integration with online utilities I was already using, such as map-tiles!® 
of satellite and geo-political imagery and reverse geo-coding.!’ This easy ability to 
communicate with remote functions, features and data comes with an additional cost 
beyond the loss of independence, self-reliance and overall stability (see Fig. 10.5). 
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Fig. 10.5 Browser based version written in JavaScript using leaflet.js showing early tests with 
meta-data extraction and visualization 


'6Map-tiles are a way of breaking large GIS-orientated datasets such as maps into smaller fragments 
that can be called and used as needed, remedying the need to load large datasets as monolithic pieces. 
A calculation is made of the geo-location that is of interest, the scale that the map is to be viewed 
and the size of the viewing port (or in our instance, web browser window); a series of request are 
made; the map-tile server and the corresponding map fragments are returned as individual files and 
then tessellated in the viewer. 

17 Reverse geo-coding is the ability to take and specific geo-location, commonly expressed in latitude 
and longitude, and return a descriptive physical world location. For example, the a reverse GeoCode 
request for the location 52.356991666666666, 1.2931251573910854 is returned as Oostpoort 12, 
Waldenlaan, Winkelcentrum Oostpoort, Amsterdam, North Holland, Netherlands, 1093NH, Nether- 
lands by the OpenStreetMap database functions that allowed the work to turn the GPS latitude and 
longitude data inside individual images into named localities with street town and country data. 
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In this instance it carried a commercial one. Although I was using a free developer 
account for one of the major mapping utility providers, the monthly usage limit was 
quickly reached for data requests. Due to license limitations on usage of the data that 
was provided, it was not legally possible to cache data between sessions of usage, nor 
store it in any long term or permanent way. This necessitated performing a reverse 
geo-code lookup for the locations of images each time the project was viewed or 
run, necessitating over 2000 calls to the remote API each time. Although the cost of 
an individual reverse geo-code request was only fractions of a cent per request, this 
added up to a cost of USD2 every time the work was viewed. The map-tile imagery 
requests came in addition to this cost. Looking at open-source alternatives became 
complex, while the service cost was free and the license terms far less restrictive in the 
data servers were not configured for heavy use, resulting in the servers banning the 
IP address of the requesting machine after the first few hundred geo-code requests. A 
strategy was developed to pre-request and cache the geo-code data and serve it from 
a self-hosted server by writing a script that requested the relevant information from 
the open-source server at a deliberately slow, throttled rate and writing the returned 
reverse lookup data to a local file in geoJSON format.'® 

This conformed with both the terms of use of the open-source server and also its 
license on use and retention of data. The second challenge was on how to deal with 
the mapping data being requested from the commercial provider. Open-source map 
data providers, understandably, were unable to supply maps data in the quantities 
the project needed and also separated the idea of individual users of the service from 
the creation of applications that would use the service. Unfortunately, there appear 
very few instances of a middle ground for artistic or non-profit uses of open-source 
services putting most artworks or projects that use third-party services or data firmly 
into this arena of being classed as commercial users and being charged accordingly. 

One of the options available within the license terms for the core corpus of map 
image data—the smaller version being in the region of 65gb of image tiles—was to 
clone the data needed and set up and operate a web-based map data server specifically 
for the project. This added a layer of technical complexity and a certain amount 
of unnecessary redundancy but afforded a break from the reliance on commercial 
providers and also added the possibility of generating a machine-specific version 
that could run as a single instance of the project with all necessary data held locally. 
This would remove the requirement for an internet connection for a greater level of 
autonomy. This involves setting part of the host computer to operate as a web server 


'8GeoJSON is an implementation of JSON (JavaScript Object Notation) that encodes geographic 
information in text form as nested objects, typically containing a latitude and longitude geom- 
etry reference optionally followed by additional data. In our instance the data encoded includes 
details of the EXIF camera data relating to images and their recording location. An example of 
typical GeoJSON entry will look similar to the following: “geometry”: “type”: “Point”, “coordi- 
nates”: [ 4.927913888888889, 52.356991666666666, 1.2931251573910854 ], “type”: “Feature”, 
“properties”: “url”: “sign6.jpg”, “date”: “2018:04:30 10:07:09”, “orientation”: 6, “make”: “Apple”, 


“model”: “iPhone SE”, “size”: 23397, “type”: “image/jpeg”, “locality”: “Oostpoort 12, Waldenlaan, 
Winkelcentrum Oostpoort, Amsterdam, North Holland, Netherlands, 1093NH, Netherlands”). 


10 Signs of Surveillance 175 


and routing web calls from the work or application back to the locally running web 
server, Sometimes called LocalHost. 

With these two elements in hand, batch generated reverse geo-code data, locally 
held, and provisioning the projects own map data server, the forming, tuning and 
final choreography of this section the piece proceeded. 


10.5 The Challenge of Live Vs Video of Activity 
or Interactivity of Exposing the Internal or Relying 
on the Surface 


In developing live digital works there is a challenge of exposing and communicating 
where the work is. In the case of paint and photography and other flat visual arts, 
the work is commonly readily identified in the surface. When working with live 
digital processes this identification, location and exposition of the work—in this case 
meaning the artistic effort, intention or focus of attention—becomes obfuscated and 
requires specialist technical knowledge to gain insight into often hidden black-box 
processes. Some artists working with digital/technological processes take the route 
of exposing the mechanism of systems, disassembling computers into their working 
component and exhibiting them exploded on gallery floors, deliberately exposing 
the ‘technology’ to indicate complexity and ‘liveness’. Another common route to 
provide this exposition to an audience is to make systems interactive or reactive 
to some aspect of the audience, making clear that something live and dynamic is 
occurring and perhaps leading the audience into consideration that which is beneath 
the surface, the ‘deeper mechanic’ inside a work or piece. With work that takes 
neither of these routes, work that is active but not interactive, that does not make 
its components part of the artwork display, the challenge is to help the audience 
understand that what they see is live and note rendered video recordings or pre-made 
files being re-played. And that each moment seen is created and becoming at that 
moment. It is being performed live, never to be repeated in that fact same way again. 


10.5.1 Exposing the Internal or Relying on the Surface 


This challenge of exposing the internal mechanism of projects and works and the 
delicate balance of the work end up being about the mechanism and the tools rather 
than the original intent is a complex one. In this work the multi-part nature of the out- 
puts from the project allows a range of expression and nuance which in another work 
would not necessarily be available. With the installation version of Signs of Surveil- 
lance the clean collection nature of the photographic source material and the exposure 
and manipulation of the image meta-data is clue enough to the live/process nature 


176 D. Buzzo 


behind the work. Alternative versions, such as the artists’ book and the controllable 
web-based version address other elements of the project sufficiently. 


10.5.2 The Final Work for Gallery Installation 


The final instantiation of the work discussed in this article is designed for viewing 
in a gallery or installation context (see Fig. 10.6), and has been tuned, sculpted and 
shaped accordingly. In an installation context the audience is mixed, with a range of 
interests, awareness and contextual insight. 

The audience, the viewer also has a range of available temporal modes within 
which to receive the work and the underlying themes and idea. Part of the work is 
paced and presented for a transitory audience with short attention and an availability 
of meaning in the surface of the work that is ‘glanceable’. That will survive artistic 
scrutiny of a glance of a few seconds. At another level the work has a deeper pace and 
rhythm that is designed to survive deeper scrutiny and draw the viewer into a deeper 
dialog with and understanding of the work, allowing nuance and interpretation at 
the pace and depth the viewer is comfortable with. The work for gallery installa- 
tion is active but unlike its open web-based counterpart (available publicly at http:// 
signsofsurveillance.com) not explicitly interactive (see Fig. 10.7). To this end, it has 
a specific set of behaviors that it exhibits to engage the viewer and draw them through 
the corpus of work. When working through the meta-data accompanying the body of 
images, a certain irony was revealed, in that, if one followed the time-stamps of the 
collected images one could re-create, with a varying range of accuracy, the route that 
I had taken through neighborhoods, cities and countries around the world over the 


Fig. 10.6 Mockup of gallery installation of early globe version 
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Fig. 10.7 Final version showing custom map styling, with clustering, image meta-data and dynamic 
transitions along the ‘walk-line’ between images over the 5 year period of collection. This location, 
Ijmeer, Amsterdam, Netherlands 


preceding five years. The time-stamp data combined with the geo-location of each 
image showed the points on the globe that I had stood at the press of every shutter 
and the capture of each photograph. Beginning to investigate these invisible lines, 
simple routines were written to create a single path through each geo-point. Using 
both Catmull-Rom!® and Bézier” curves different paths were visualized showing 
the path between each captured image. 


10.6 Conclusion 


In this chapter I have attempted to expose and discuss some of the approaches and 
practices that I have evolved and adopted over the last 30+ years of working with 
digital technologies. My original thoughts were that working in conjunction with 
complex, highly sophisticated tools and systems which transformed the relationship 
from one of artist and tool into much more of an intimate discussion, a collaboration 


19 A Catmull—Rom spline, named for Edwin Catmull and Raphael Rom, is a special case of a cardinal 
spline. Catmull—Rom splines are frequently used to get smooth transitions between key points and 
are often used in animation and similar computer graphic works. 

20Named after Pierre Bézier, a Bézier curve is a parametric curve used in computer graphics. Several 
points using Bézier curves can be combined to form a Bézier spline. 
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where one asked questions of the systems one engages in, navigating and negotiat- 
ing the parameters and avenues in and through the conceptual structures, functions 
and processes inside the machine. From this comes the innate understanding of the 
systems and materials one is working with, what I have called semantic affordances. 
Sadly, this conceptual flexibility and strength that digital materials bring also brings 
multiple points of failure, usually with no graceful decline. A point of failure at any 
juncture in a digital artwork is usually a complete failure. This is what I mean when 
I talk of the brittleness of digital things. For this work, which extensively involved 
with the collection of a large body of individual items working in the digital gives 
great flexibility to collate, manipulate, transform and express. It also gives myriad 
opportunities for instantiation of a work in the physical world, and huge availabil- 
ity of choices of how to make an intervention into the mind of the viewer. At this 
instance, the challenge came in making gallery-orientated versions and web-based 
while using web technologies and remote data in both. The issue of reliance on, and 
of cost, when using external systems, such as map-tile data and geo-location services 
was a particular challenge but let to new and interesting areas setting up self-hosted 
and local versions of some of these external commercial data sources. This does, 
however, have its own drawbacks, and the writing discusses in some detail the philo- 
sophical, cultural and ultimately ontological struggle when engaging with external 
systems, APIs and data. Also that, they have their own political approach, their own 
ontologies and that can be a hegemonist force with the potential to overwhelm the 
internal ontology that an individual work of art generates for itself. This question of 
the internal complexity and dialog that goes on inside active digital works is the last 
point in this writing, investigating some of the challenges that are balanced when 
dealing with the decision of how much to expose the internal structures of a work 
to bring viewers deeper inside, drawing them below the surface of a work while 
not making the work purely about the internal structure. Thus, working to retain the 
original focus of the work but helping give insight into the live process at work at 
the moment of performance, particularly with works with any generative nature to 
them. 


10.6.1 Final Reflection 


The final version of this work, or at least the current version, is part of a body of pieces 
that are differing expressions of a formalized process of collecting and collating. This 
collecting began as a dérive of observation and photographic collecting around the 
world. Begun initially as a completely non-verbal process the project developed its 
own vocabulary and logic over the years of photographing, to the point where the 
material began to guide the form and the final work. Only then began the process 
of exploration in the studio, of applying technological processes to illustrate and 
instantiate the mood developing inside of the work. This process is common to much 
of my work and my day-to-day artistic activity: one that is an ongoing dialog I 
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have with the world, of observing, looking, making, and underneath it all ultimately 
seeking sense-making. 

The work, while not forcing a specific political point, is intimately political. It is 
intended as an exposure of pervasive surveillance of our physical world, to a scale 
that it continues to surprise me, even as I become inured to it. This journey we have 
chosen to take with surveillance technology, and the ability we have built to track, 
record and intrude into every aspect of the body public, and our individual bodies, 
private is operating at a scale that many would find unimaginable should they be able 
to see it clearly. 

As Noam Chomsky, never one to use an alarming phrase when calm reasoned 
words will do, says [11]: 


there are two things to bear in mind, one is that the phenomenon (of surveillance) should not 
be at all surprising, the second is that the scale of it is surprising. 


10.7 Epitaph for a Digital Artwork 


I sit and write, longhand on paper with a pen, the epitaph at the closing of this chapter. 
The document and journey written covers six countries, countless hours and edits, 
full of insight and reflection. The final edits are yet to be done but what I can write 
now is the epitaph. Reading back through this writing, I find I have made a fuller, 
more complete explanation of my internal process, discussion and artistic activity 
than ever before in one place. At times on technical, philosophical and political, I 
trust the reader will forgive me streams of consciousness and occasional melodrama 
of the miniscule internal theatre that is the struggle of working with digital tools. This 
writing, alongside ongoing development, coding and versioning of the software, has 
given me space to reflect on process and practice with a fresh perspective. Thinking 
of the work, in progress for nearly half a decade, makes me reflect that most readers 
of this will never see it. That by the time this volume is typeset, proofed, approved, 
printed and distributed, the work may, in practical terms, cease to exist. Web domains 
I pay for will expire. Servers will be upgraded, patched, obsoleted and retired and 
I will go on to new projects, works, technology and software. In very real terms 
this chapter of ink-on-paper (or the undoubtedly shorter-lived pixels-on-screen) will 
outlast the work by a magnitude of scale. Sitting in Hong Kong in late June 2019, 
the mood is somber but resolved and the world seems fragile and tenuous. This work 
on Signs of Surveillance is a small postcard in a discussion of technological change, 
reflecting on the fundamental importance of the ephemerality of art, of the importance 
of its functionless-ness. On the innate lack of practicality that those versions and 
instantiations of it that require upkeep, maintenance and care to continue to exist are 
happily always consigned to memory. This is the epitaph to a work of art that will 
soon cease to be. That will fail by nature of its powerful but short-lived reach and 
the fickle brittleness of the substance of which it is formed and the un-real substrate 
on which it is built. 
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As Picasso, Paul Vale’ry or Da Vinci may have said?! 


Art is never finished, it is only ever abandoned. 


and the abandonment of digital works into a continually changing virtual world 


affords them a very short lifespan when left to fend for themselves. 
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Abstract Marlena Novak (visual artist) and Jay Alan Yim (composer/sound artist), 
under the collaborative name localStyle, have been making work that addresses 
environmental and socio-political concerns through the use of a wide range of media 
since 2000, particularly utilizing technological tools in their creative practice. Deep 
interest in understanding the contemporary conditions of non-humans and humans 
is informed by their engagement with publications and texts, attending, participat- 
ing in, and hosting talks, panels, and screenings, and meeting with specialists and 
scientists regarding the topics of their research. Choral is a twelve-minute audiovi- 
sual installation commissioned by 150 Media Stream for a sculpturally unique LED 
panel display. This chapter describes the genesis of the project, from its inception 
as an artistic response to news reports of widespread coral bleaching events compli- 
cated by climate change, through a concerted phase to properly ground the project 
in scientific research, followed by an overview of the methodologies and resources 
used to realize the project, and including a discussion of the technical and conceptual 
challenges posed by working on a very large scale display medium in a public venue. 
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11.1 Introduction: Corals Are in Crisis 


Many coral species cannot adapt quickly enough to warming ocean temperatures and 
increasing acidification, threatening reefs worldwide and creating the potential for a 
catastrophic loss. The habitat created by coral reefs makes them fundamental to the 
sustainable diversity of ocean fauna, and to all living things—including humans— 
that rely upon this part of the earth’s ecology. 

Our audiovisual installation regards coral reefs as the ‘voice’ of the Anthropocene, 
hence the title Choral. Reefs play a foundational role in providing habitat for a 
quarter of all marine species and these ecosystems are in crisis. Although they face 
challenges from multiple directions, we are approaching our piece from an optimistic 
perspective. Techniques for fostering recovery and regeneration are already under 
development: a dedicated part of the scientific community is determined to find ways 
to make these methods scalable, and this is cause for hope. 

Through our creative work, we endeavor to join this collective effort by con- 
tributing to a larger-scale human response. Although scientific research informs our 
fundamental approach to the visual appearance of various coral species, the way 
corals behave in the work reflects our shared imagination through speculative under- 
water world-building—rather than emulating a literal documentary. We were invited 
in Spring 2017 by the curator for 150 Media Stream [1], Yuge Zhou [2], to make a 
moving image artwork for 150’s unique wave-shaped 89 vertical panel LED-screen 
lobby display. This custom-designed technology allows for working at a large scale— 
the Media Stream display measures 6.5 by 47 m—while the nature of this venue in 
the heart of downtown Chicago allows for a larger audience to encounter and con- 
sider this topic. Our response, Choral, is a computer-generated 3D animation with 
electronic sound that addresses our inquiry focused on the ecosystem of coral reefs 
(Fig. 11.1). 


11.2 Initial Research Phase 


Having attended Ars Electronica for five years, three Documentas, several Venice 
Biennales, as well as having participated in the STRP Festival, the National Art 
Museum of China’s TransLife Triennial, the Taipei Digital Art Festival, and encoun- 
tering a wide range of art installations incorporating sound, moving image resources, 
or combinations of the two, in addition to maintaining a collaborative praxis as artists, 
both of us are professors who have taught numerous classes since 2003 that involve 
new media and installations. We take our responsibilities to stay well informed seri- 
ously as it is integral to our pedagogical activities. The following essay may best 
be understood as intentionally circumscribed in its scope, since a more expansive 
survey of digital installations as a genre would exceed our remit. 

As artists we have been fortunate to have a range of public platforms available 
to reach a diverse audience. The concern for what we, as a species, have done to 
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Fig. 11.1 Choral at 150 N Riverside Plaza, Chicago Illinois, USA. Photo Michael Salisbury [3], 
image copyright ©Michael Salisbury and reproduced by permission 


the planet is reflected in our work as individual pieces are used as opportunities to 
probe deeper into these topics while assessing the effectiveness of different modes of 
expression on audience engagement. Given our history of collaborating on projects— 
prick (2006-08) [4], scale (2009-10) [5], Bird (2011-14) [6] —that involved aspects 
of what two physicist colleagues have termed “cultural processing” [7], we felt it 
necessary to delve into more informed research beyond our initial encounters with 
journalistic reports dating from 2014 regarding the catastrophic events that coral reefs 
were suffering around the planet. Determined to gain a better grasp of the situation, 
we first undertook a program of self-education through a variety of documentary 
films and videos, paralleled by perusing a selection of more recent scientific papers. 

In the course of learning more about the multiple causes and disastrous conse- 
quences of coral bleaching and death, we encountered the research program of two 
Chicago-based scientists, Luisa Marcelino (Research Assistant Professor of Civil 
and Environmental Engineering, Northwestern University) [8] and Timothy Swain 
(Postdoctoral Fellow, Northwestern University and the Field Museum) [9], whose 
published articles were a source of inspiration to us, and who agreed to meet us in 
person. Prior to our March 2018 meeting, we viewed a WTTW television segment 
where both were interviewed and we were struck by Dr. Swain’s observation, “The 
coral reef ecosystem may be the first ecosystem lost to climate change” [10]. Though 
the topic was initially grim, our discussion with Marcelino and Swain was infor- 
mative and productive. An afternoon conversation with these scientists absolutely 
validated our concerns regarding the critical condition of the global ecosystem of the 
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reef and our commitment to developing the project for 150 Media Stream. We learned 
that the Marcelino Research Group’s work at Northwestern points to coral polyps as 
architects, wherein the fractal properties of their skeletons redistribute light as a pho- 
tosynthetic resource within the colony. Further study of this phenomenon suggests a 
relationship between faster-growing species and increased vulnerability to bleaching. 

At the end of our meeting, Prof. Marcelino offered to curate a number of critical 
papers for us to study and Dr. Swain invited us to join him at an upcoming visit to 
Chicago’s Shedd Aquarium to observe some of the coral works that he was engaged 
with in their laboratory. Having a chance to see these animals close-up and in-person 
brought a more immediate level of appreciation of their charismatic qualities. It was 
moving to see how tiny these creatures were while playing such a large role in marine 
ecosystem sustainability and by extension human economic dependency on that very 
ecosystem. As Rebecca Albright says: 


Although reefs cover just 0.1 percent of the ocean floor, they support nearly 25 percent of all 
marine species, including fisheries that feed millions of people worldwide. They also provide 
natural breakwaters that protect coastal communities by reducing wave energy by up to 97 
percent and wave height by up to 84 percent. And they generate vast tourism revenue. If we 
lose reefs, we jeopardize the livelihoods of 500 million people and more than $30 billion 
annually in goods and services [11]. 


Many people benefit unknowingly from the coral reef habitat; numerous articles 
refer to the reefs as the “rainforests of the sea” [12] or “the medicine cabinets of the 
21 century” [13], from whence a number of anticancer, antiviral, and pain manage- 
ment drugs are being explored, as well as bone graft material. During our behind- 
the-scenes encounter at the Shedd Aquarium, one of the staff members remarked 
that although they had valuable and instructive coral displays, they felt they weren’t 
reaching enough of a diverse public to get their message across, and Swain replied: 
“That’s what Marlena and Jay do, as artists’. Consequently, we realized Choral 
could play a role beyond expressing our personal commitment to Anthropocene 
issues; it might also have the potential to raise public awareness and connect that 
awareness to the marine biology community that is actively working on solutions to 
the problems. Helping this community become more visible might ultimately trans- 
late into the public policy decisions we collectively need to be enacted for corals 
to be saved. One of the papers Marcelino referred us to was “Coral Reefs in the 
Anthropocene” [14]. Among the noteworthy points found: 


“[...] research has revealed provocative complexities in the expanding knowledge base about 
corals and their ecosystems, thus stimulating our imaginations regarding alternate ways to 
envision the future of reefs” and “Increasingly, coral reef scientists and managers encounter 
previously unseen configurations of species” [14] 


According to Terry Hughes, the reality is that: 


reef ecosystems are more dynamic and patchier, as well as increasingly different to any- 
thing that people have encountered before. Embracing this paradigm shift will necessitate a 
transformation in the governance and management of these high-diversity ecosystems [14]. 


There won’t be any single answer to these problems; solutions to fostering sustain- 
ability for corals in the future could build upon a wide range of methods developed 
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by Marcelino’s group as well as many other scientists working in the field. Some of 
these include the creation of a meta-level tool to correlate bleaching data from all 
previous coral studies into a coherent worldwide index [15] and identifying species 
better adapted to survive. We learned from Albright’s article that: 


Mary Hagedorn of the Smithsonian Conservation Biology Institute has established the first 
genome repository for endangered coral species. [...] Her team has developed a freezing 
system for sperm that can be applied to a wide range of coral species. To date, the team has 
successfully banked 16 species from around the world (2 percent of the earth’s estimated 
800 species) [11]. 


Our optimism increased when we learned about the work of the late Dr. Ruth Gates 
at the Hawaii Institute of Marine Biology and her collaboration with Madeleine van 
Oppen of the Australian Institute of Marine Science to develop next-generation super- 
corals. Their labs are working on selective breeding and epigenetic tuning to tolerate 
warmer sea temperatures, and selective breeding of the symbiotic zooxanthellae and 
the bacteria that comprise the microbiome of coral polyps to breed more robust 
colonies. SECORE International is working on both increasing genetic diversity and 
fostering the odds in favor of coral larvae surviving past their initial vulnerability to 
being eaten by other marine creatures; when they are big enough to be outplanted, 
these new generations of young corals can help repopulate damaged reefs. One of 
the best designs for an economically scalable substrate for baby corals to grow upon 
is a grapefruit-sized concrete tetrapod, textured so that the larvae can readily attach 
themselves and start their reef-building. These tetrapods are featured in one sequence 
of Choral as a symbolic representation of the collective efforts of humans to channel 
their ingenuity toward finding a solution to the bleaching and reef degradation crisis. 
(One of our contacts at the Shedd Aquarium, Mark Schick is a collaborator with 
SECORE in the tetrapod project.) Other, more speculative approaches under devel- 
opment include deploying groups of underwater robots that would re-cement broken 
coral fragments to the deep-sea cold-water reefs they came from, aiding re-growth. 

Having first-hand discussions with scientists who are dedicated to identifying and 
addressing the complex issues that promote coral bleaching was encouraging. These 
experiences bolstered our decision to create an audiovisual artwork that engendered 
hope and fostered public engagement rather than despair and cynicism. A meeting 
with Hobson [16] who is aresearcher at NORC was also valuable to our project; draw- 
ing on personal experiences from research projects undertaken when she was based 
in Australia, Hobson told us that “a noisy reef is a healthy reef” which inspired us to 
investigate this aspect and subsequently became a factor in shaping the conceptual 
approach to sound design and soundtrack orchestration in Choral. 


11.3 Project Development, Phase Two 


A steady intake of information fostered the necessary processes of refinement and 
metamorphosis, guiding the project’s vector away from didacticism toward a more 
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poetic direction. A project of this scale compelled us to undertake two different, 
though complementary research tracks. The scientific data obviously had to be assim- 
ilated and understood to inform our conceptualization of the project, but we also 
needed to comprehend what kinds of hardware, software, and personnel resources 
were needed to realize our vision, and to develop a budget and a team that could 
feasibly support that goal. With the scientific side in full swing by the onset of 
winter 2017, we accepted an invitation to participate in the Species/Biodiversity 
Loss panel of the conference Why Do Animal Studies?: The Turn to the Quasi-, 
Post-, Anti-, Non-, Para- [17] in April 2018, with our presentation “Choral, a work 
in progress”. At minimum, it was a chance to get expert feedback from scholars 
whose critical thinking and writing on non-human others and Anthropocene-centric 
issues had been impactful on our own mode of inquiry and efforts. As is common in 
these types of conferences, it was also a remarkable opportunity to be immersed in 
the multidisciplinary confluence of scholarly and artistic projects that comprise the 
field. 

One of the strengths of how localStyle’s collaborative approach has deepened since 
its founding in 2000 is the way we are able to pursue independent development tracks 
in our working methods based on our individual skill sets and areas of expertise, while 
maintaining a constant conceptually driven dialogue that revolves around the central 
thrust of any specific project. Inspired by the hospitality of corals to create habitats for 
other creatures, Novak began modeling, texturing, and animating individual corals— 
devoting time to both individual colonies from various species as well as to detailed 
hexacoral polyps—and to assemble a synthesized version of their biome utilizing 
3D software tools (Autodesk Maya, Arnold Renderer, Pixologic zBrush) on a Mac 
Pro workstation. In the conference, her slides incorporated digitally created images 
that symbolically represented coral bleaching and the degradation of the ecosystem 
(Figs. 11.2 and 11.3). Based on audience feedback, this imagery turned out to be 
effective for attendees in the academic context of various presentations zeroing in on 
challenges that ranged from justice, law, and ethics to postcolonialism to disruption 
on a variety of planetary scales. These preliminary images thus played a crucial role 
in our conceptual development of Choral, even though we ultimately decided to 
tactically focus the installation’s mode of address toward one that emphasized the 


Fig. 11.2 Study for Choral: Reef Ecosystem Degradation. 3D image: Marlena Novak, image 
copyright ©M. Novak 2019 and reproduced by permission 
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Fig. 11.3 Study for Choral: Zooxanthellae expulsion. 3D image: Marlena Novak, image copyright 
©M. Novak 2019 and reproduced by permission. 


hopeful side of human efforts to rescue coral. As with any environmental or climate- 
related issue, the presence of the political dimension is never far from the surface, 
and the operational question arises as to how people can be motivated to engage and 
empathize with a topic so as to enable policy-making to occur in positive ways. 

Shortly after meeting with the Northwestern University scientists, the project 
implementation side of Choral began to accelerate. The organization that devel- 
oped for the installation at 150 N Riverside Plaza placed Novak as co-producer, 
director, project manager, lead 3D visual artist (concept, modeling, texturing, and 
animation), and motion graphics conceptual developer; Yim was the co-producer and 
composer/sound designer as well as being involved with the 3D and motion graphics 
conceptual development. Clearly a work of this scope would have much to gain from 
having project assistants; they would benefit from the experience, contribute to the 
work in a multitude of ways, and be credited for their roles in a high-profile project. 

In Spring and Fall 2018, Novak assembled a group of young artists who had 
formerly studied with her at the School of the Art Institute of Chicago. At an early 
stage, she compiled a collection of references as an encapsulated database to share 
with team members. These comprised science-based pieces of journalism in print, 
web browser, and video formats, organized into three main groups: 


(1) how corals live, 
(2) reasons for humans to care about corals, and 
(3) factors threatening their existence. 


The first group of references concerned coral taxonomy, structure, anatomy, habi- 
tat, behavior, etcetera. It was important for everyone on board to assimilate some of 
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the basic information about these animals: coral polyps have a symbiotic relation- 
ship with photosynthetic dinoflagellates; their survival depends on these algae that 
provide between 90 and 98% of their nourishment; these zooxanthellae live under 
the epidermis of each polyp; the polyp’s epidermis is coated with bacteria; and these 
three organisms appear to benefit one another. Especially pertinent were key points 
provided by the Coral Reef Alliance: 


Coral reefs are often referred to as the medicine chests of the sea. A number of creatures found 
on reefs produce chemical compounds that have been isolated for human applications—and 
many more are yet to be discovered. Scientists have developed treatments for cardiovascular 
diseases, ulcers, leukemia, lymphoma, and skin cancer, all from chemicals in reef plants 
and animals. Other compounds reduce inflammation, kill viruses, and relax muscles. The 
beautiful and fragile creatures of our coral reefs have the potential to make even greater 
contributions to our lives by providing new cures for life-threatening diseases. More than 
half of all new cancer drug research focuses on marine organisms. In addition, coral’s unique 
skeletal structure has been used to make our most advanced forms of bone-grafting materials 
[18]. 


In addition to the stresses of temperature-induced bleaching and ocean acidifi- 
cation weakening their skeletal structures, other factors (many of which are under 
direct human control) combine to threaten coral’s existence: in the natural competi- 
tion between coral and seaweed, warmer water combined with fertilizer run-off gives 
seaweed and other macro-algae the advantage; overfishing of species that ordinarily 
keep seaweed in check by feeding on it exacerbates the stress; governmental policies 
that revoke environmental protections for existing marine national monuments result 
in more pollution from industrial exploitation such as mining and oil development; 
even tourism can be a factor as products like many sunscreens are toxic when intro- 
duced to the water. (The most recent evidence points to these products also being 
toxic to humans.) 

As work progressed, we held group screenings of selected documentaries to har- 
monize our perspectives. Joining localStyle in preproduction concept development 
were Sally Jo [19] and Malu Ayers [20], and project intern Shinuo Snow Xu [21]. 
As team assignments solidified, Ayers and Xu worked as 3D artists (modeling, tex- 
turing, lighting, animation); Max Crider [22] built and animated 3D rigs for polyp 
and coral animation in the close-up scenes. Nathaniel Gillette [23] joined us in the 
Fall as a 3D and 2D artist and became motion graphics coordinator. Also in the Fall, 
Mak Hepler-Gonzalez [24] joined the Choral team as assistant producer, rendering 
manager and 3D artist involved in all aspects of 3D production, along with post- 
production, conceptual, and technical development; he contributed several advanced 
3D techniques for compositional layout, rendering, and animation that would serve 
the project until its completion (Fig. 11.4). 


11.4 Workflow 


Due to having an artist’s residency abroad during some of this period, we set up a 
shared online project folder and a blog to stay involved with each other’s progress; 
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Fig. 11.4 Choral team members working at Unspecified Research Lab (top to bottom: Gillette, 
Novak, Hepler-Gonzalez, Ayers, Xu) Photo Jay Alan Yim, image copyright ©J. A. Yim 2019 and 
reproduced by permission 


we also scheduled regular Skype meetings while away. Once back in Chicago, studio 
sessions resumed. Our working method evolved to include collaboration on many 
levels: in some cases, a single artist modeled, textured, lit, and animated their models; 
at other times, they would hand off models they made for another team member to 
texture, while another artist did the lighting and another executed the animation. 
If one of us developed a new material that would be suitable for others to use, or 
a lighting-set, or an animation technique that would be effective for each other’s 
models, we shared these 3D elements. For the most part, our roles were fluid, unlike 
production hierarchies where a single person works on modeling and another solely 
engages with texturing, and so forth. Except for one artist who was only involved 
with much of the rigging, we exchanged and reversed the majority of our roles. This 
created a special sense of team effort and connection to the final piece. When one of 
our valued members relocated to another part of the country, we continued to work 
together via retrieval and development of their uploaded files. Choral would not exist 
in its current state if not for the synergy of this solid collaboration. 

We were fortunate to receive supplementary grant funding from the Illinois Arts 
Council and a Faculty Enrichment Grant (SAIC) that supported aspects of the project; 
several new computers were necessary to address the rendering time needed for the 
unusually high resolution of each frame of animation. In addition to the previously 
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mentioned software programs (Maya, Arnold, zBrush), Allegorithmic Substance 
Painter was used for texturing and Adobe After Effects was used for composit- 
ing. Over the course of the project, two custom-configured PC workstations were 
acquired by Unspecified Research Lab, which is localStyle’s studio facility. Sound 
design/scoring took place on two Mac workstations. Throughout the project, as many 
as eight laptops were used in conjunction with the desktop machines. Work continued 
in a mixed Mac/Windows environment, tailored to take advantage of the capabilities 
of each platform (Fig. 11.5). 

Fortunately, we had designed our studio space to be flexible in usage, which 
allowed a number of artists to work together comfortably and in close proximity to 
the kitchen and seating areas where we could take breaks as necessary. We regularly 
provided lunches and dinners, and team members ate together depending on the 
work periods (Fig. 11.6). Sharing home-cooked meals, often from items grown in 
our garden, built a closer sense of community; several times an artist from the team 
would also contribute cookies or bread they had baked. Group meal conversations 
sometimes addressed recent project achievements or trouble-shooting concerns, but 
mainly it was a time to focus on positive topics. 


Fig. 11.5 Choral team members working at Unspecified Research Lab (left to right: Yim, Ayers, 
Hepler-Gonzalez, Gillette, Crider). Photo Marlena Novak, image copyright ©M. Novak 2019 and 
reproduced by permission 
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Fig. 11.6 Choral team members enjoying a lunch break (clockwise from upper right: Yim, Gillette, 
Xu, Hepler-Gonzalez, Ayers). Photo Marlena Novak, image copyright ©M. Novak 2019 and 
reproduced by permission 


11.5 Compositional Design and Technical Challenges 


There were a number of design challenges unique to this project. One of them entailed 
the sheer physical size of the video display; with the tallest panels standing at nearly 
22 feet and the full array spanning 153 feet in width, we needed to make the visual 
elements narratively coherent even though it is impossible to stand back far enough 
in the lobby to see all of the LED blades simultaneously from the center. It is quite 
a different experience to visually compose a scene with digital models being made, 
textured, and animated on 28-in. computer screens having 3840 x 2160 resolution; 
we could assess the design in its entirety. Though we are fortunate to have a 4.9 m 
(diagonal) projection screen in-house at URL, that is still dwarfed by the size of the 
tallest blades (6.54 x 0.225 m) at 150 Media Stream. The physical scale impacted 
team decisions regarding formal qualities such as repetition, proximity, motion, and 
pacing, requiring care as to where visual elements would be located to create the most 
dynamic mise-en-scène. Strategically, the array’s width helped direct the temporal 
proportions of different passages in Choral, as the situation compelled the viewer to 
move actively through the space to take in as many CGI corals as possible. 

The second major factor in the design was working with the negative space that 
is an integral component of the display: we had to be conscious of this site-specific 
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parameter to ensure that animated elements retained maximal legibility as they nego- 
tiated the voids between panels. Though it was quite useful to have a TIFF mask 
composited as a layer to preview image sequences on our computers, we appreciated 
the difference it made to see and hear the draft versions of Choral in situ. In a sense, 
it was visually somewhat like working with a reverse zoetrope in that the alternating 
slits and opacities worked together to create an interrupted continuity. 

LED display resolution was a corollary aspect of the size factor: 150 Media 
Stream’s total pixel count was 15360 x 2160, the equivalent of four 4K screens 
arranged in a horizontal panoramic view. Rendering moving image sequences at 
high resolutions is intensive in terms of both time and computational resources. 
We had non-exclusive access to a multiblade render farm, and non-exclusive access 
to several banks of computers, in addition to the machines at URL. Emily Kuehn 
[25]—who is an exceptionally resourceful staff colleague at SAIC—provided crucial 
support and assistance in addressing the unique and precise Maya template for the 89 
screens (which measure a variety of heights and widths), and optimizing the render 
sequencing setup for the 16K display. 

The acoustics of the building conditioned the approach taken to orchestrating the 
electronic soundtrack and equalizing the mix: the presence of many hard, reflective 
surfaces—stone, glass, metal—and the height of the atrium effectively disperses 
much of the audio signal and makes it nonlocalizable. Architectural design at 150 N 
Riverside Plaza produces acoustic results that are comparable in some significant 
ways with the reverberant spaces often associated with the performance of large 
choral works. This made the decision to base the soundtrack on Thomas Tallis’ forty 
voice motet [26] an apposite one, both for these acoustic characteristics and because 
the eight-channel audio system corresponded to the polychoral division of voices in 
his score into eight separate vocal quintets. Additional motivations for choosing this 
point of departure will be discussed below. 


11.6 Choral Animation Summary 


The introduction begins with waves on a dark sea (where the ocean is a digital 
simulation). In the tradition of an origin myth, we see friendly bioluminescent polyps 
ascend to take their places amidst a field of stars in a night sky filled with simulated 
constellations (Fig. 11.7). 

Stylized bubbles rise against the water to become stars; many are replaced by the 
astral polyps. As the polyps wink out, the celestial field metamorphoses into a digital 
wire frame that descends to form the contours of a seabed, gradually evolving into 
more solid terrain (Fig. 11.8). 

As an aqueous curtain rises, a richly populated reef full of many species of coral 
is revealed, and the virtual camera pans at a deliberate pace across the ocean floor 
(Fig. 11.9). 

Being digitally modeled, our reef places all of the corals front and center to empha- 
size their importance to the community of fish, other creatures, and marine plants that 
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Fig. 11.7 Polyps rising. Photo Michael Salisbury, image copyright ©Michael Salisbury and 
reproduced by permission 


Fig. 11.8 Celestial polyps Photo credit: Michael Salisbury, image copyright ©Michael Salisbury 
and reproduced by permission 
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Fig. 11.9 Seabed revealed. Photo Marlena Novak, image copyright ©M. Novak 2019 and 
reproduced by permission 


would comprise and inhabit this ecosystem in nature. Throughout our 3D animation, 
the corals are singing when visible (Fig. 11.10). 

The digital camera zooms in and slowly circles around several of the more charis- 
matic colonies on this imaginary reef. Despite most coral polyps being individually 
very small animals, in the aggregate, they play an outsized role in the formation 
of reef ecosystems, which is our motivation for modeling them in detail for these 
close-up views (Figs. 11.11 and 11.12). 

The intended goal of these two sequences is to engage audience members with the 
awe-inspiring beauty of coral: to reconnect them to personal memories of experienc- 
ing coral in the sea or other environments. Hearing viewers remark, “We should go 
back to the Shedd Aquarium” after seeing our piece in the lobby of 150 N Riverside 
underscored the importance of making this kind of contact since Chicago’s Shedd 
Aquarium has very good coral exhibitions with detailed information regarding the 
stresses on coral ecosystems and how those risks will impact our lives if corals are 
allowed to perish. 

The following passage features many computer-generated tetrapods with growing 
baby corals descending through the water (Figs. 11.13 and 11.14). These represent 
the small concrete forms that researchers have designed to serve as ideal substrates 
for newly attached larvae to grow upon. Outplanting tetrapods is one of several 
complementary solutions currently under development and the Shedd is involved in 
promoting this approach. As artists, we had hoped that an engaged public would 
be curious to know more about what the tetrapods are and how they relate to coral 
reefs; during the test screenings, we have been pleased that this sequence has sparked 
people’s interest and initiated conversation. 
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Fig. 11.10 Coral reef Photo Marlena Novak, image copyright ©M. Novak 2019 and reproduced 
by permission 


Next, the scene crossfades from objective potential to the metaphysical, as the 
corals in Choral dance and demonstrate choreographic prowess and graceful coor- 
dination. They use their bioluminescent qualities in place of costumes. The dance 
aspires to usher in their miraculous spawning event (Figs. 11.15 and 11.16). 

Finally, the magical moment arrives that happens once per year when coral 
colonies spawn in synchronization with other colonies of the same species: eggs 
and sperm are released by the millions to join in the water and produce new coral 
larvae. For those species whose preference is to incubate their larvae inside the bod- 
ies of their polyps, the mother polyps shelter their eggs and are fertilized by sperm 
from male polyps; when the time comes, the larvae emerge from the mouths of their 
mothers. Regardless of the method, the new corals and gametes rise in the water col- 
umn toward the starry sky, swimming up to the surface as they are attracted to light, 
and eventually falling back to the seabed where they will begin new colonies. Sym- 
bolizing the cycle of regeneration, our animation is formulated to seamlessly loop 
from the spawning sequence back to the ocean scene at the beginning (Figs. 11.17 
and 11.18). 
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Fig. 11.11 Coral close-up view. Photo Jay Alan Yim, image copyright ©J. A. Yim 2019 and 
reproduced by permission 


11.7 Sound Design and Electronic Score Summary 


A healthy reef is in reality a noisy place, thus, our narrative begins with a rhythmicized 
evocation of that percussive sound world, gradually superseded by more sustained 
and contemplative textures. When the seabed materializes and we see the reef replete 
with coral colonies of many kinds, we hear the polyps singing polyphonically; it is 
left to the viewer whether one chooses to interpret this as metaphor or fantasy: that 
corals are the voice of the Anthropocene. 

These non-human choristers are constantly present in the sound design of Choral 
until the final gamete sequence fades back into the ocean scene of the opening 
frames. Conceptually, the soundtrack began as a recomposition project, taking one 
of the most celebrated choral works in the early music repertoire, Thomas Tallis’ 
1570s motet for forty voices “Spem in alium”, and applying an idiosyncratic version 
of musica ficta [27] so pervasively as to relocate the harmonic world of the motet 
to that of the present day. (At this point, it may be worth noting that this author 
first encountered the Tallis score as an undergraduate composition student in 1977. 
It was immediately elevated to my personal list of desert-island discs, and from the 
time of initially becoming a professor of composition, has for decades since then 
been recommended as a study score to my own students. It is truly a marvel of 
counterpoint and spatialisation, arguably superior—particularly in terms of melodic 
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Fig. 11.12 Active polyp close-up. Photo Marlena Novak, image copyright ©M. Novak 2019 and 
reproduced by permission 


independence—to Alessandro Striggio’s [28] own forty voice motet, “Ecce beatam 
lucem”, which is typically seen as the precursor to and inspiration for the Tallis.) The 
timbral world of the music is similarly updated to incorporate otherworldly vocal 
sounds with electronic textures far removed from the sensibilities of the Renaissance. 
In the spirit of technical inversion, all of the percussive sounds are triggered by the 
rhythms of Tallis’ motet, whereas the smooth resonances that create the impression 
of being in the ocean have been fashioned from underwater hydrophone recordings 
of reefs, which are full of clicking sounds with short attack and decay times. 

A variety of software tools contributed to the workflow: Finale for MIDI score 
input and recomposition in a staff notation environment (due to my classical com- 
positional training, staff notation offers more effective control over counterpoint 
and harmony than the piano roll display in Live); Ableton Live as a MIDI track 
host for softsynth instrument playback and recording (including Native Instruments’ 
Absynth 5); Sound Forge for time compression and as a plugin host for DSP and sound 
design (transforming the hydrophone recordings, shaping samples for the descending 
sounds in the tetrapod scene, spectral blurring of the vocal textures); Audacity for 
timestretching; and ProTools for overall track assembly, synchronization with video, 
and mixing. In the plugin department, Eventide’s Blackhole reverb and Unfiltered 
Audio’s BYOME were invaluable. 
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Fig. 11.13 Tetrapod with growing coral close-up. Photo Jay Alan Yim, image copyright ©J. A. 
Yim 2019 and reproduced by permission 


Fig. 11.14 Tetrapods descending to ocean floor. Photo Michael Salisbury, image copyright 
©Michael Salisbury and reproduced by permission 
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Dance sequence close-up. Photo Marlena Novak. 


Fig. 11.15 


and reproduced by permission 
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Fig. 11.16 Dance sequence, long view. Photo Jay Alan Yim, image copyright ©J. A. Yim 2019 
and reproduced by permission 


Fig. 11.17 Spawning coral sequence close-up. Photo Marlena Novak, image copyright OM. Novak 
2019 and reproduced by permission 
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Fig. 11.18 Spawning coral sequence long view. Photo Jay Alan Yim, image copyright ©J. A. Yim 
2019 and reproduced by permission 


The first (and most intensive) step was undertaking recomposition of Tallis’ motet 
in Finale. Working in score view gave me the greatest degree of control as chromatic 
modifications—with increasing deviations—were made to the original score. Garri- 
tan Orchestra samples were used to record the basic choral textures, but these were 
modified substantially in Sound Forge with granular spectral tools and other audio 
processors. The MIDI data from the Finale file was exported to Live for further 
development. In Live, the combined MIDI data for all forty voices of the complete 
motet were used as a single clip to trigger a variety of softsynth instruments; each was 
recorded and rendered as an 8.5 minute audio track. These tracks were synchronized 
to the same start time in ProTools because the scoring strategy was somewhat uncon- 
ventional: (1) to have the recomposed motet play uninterruptedly from beginning 
to end, thus ensuring continuity, while (2) the electronic orchestration changed for 
each animation sequence in order to best support the scene, via automation curves 
when making the final mix. MIDI data was processed using some of the commonly 
available utilities found in most modern DAWs: automation of pitch bend data made 
the microtonal retuning of the harmonic series arpeggios in the final scene easily 
achievable and straightforward, and a moderately complex deployment of an arpeg- 
giator function in the techno-prog scene was used to create the swirling melodic 
arabesques. 
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11.8 Conclusions 


Exploring this topic through the use of computer-generated motion graphics and 3D 
software has created an opportunity for us, along with our artist team members, to 
look very closely at these remarkable animals in order to observe their attributes as 
best as possible. But it didn’t stop at digital mimicry; using these tools compelled a 
type of looking that resulted in awestruck wonder at the complexity and beauty of 
coral’s functions. The privilege of having this enormous, custom-designed ultra-high 
resolution LED display to present our project in a public space allows a wide audience 
of diverse means and pursuits to engage with this subject. We hope that some portion 
of the viewers will be motivated to consider what role they may themselves play in 
fostering the sustainability of coral reefs, as scientists in the field have made it clear 
that we must act now without hesitation to save this ecosystem from extinction. 
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Chapter 12 A) 
Cyberdreams: Visualizing Music get 
in Extended Reality 


Jonathan Weinel 


Abstract From the visual music films of the twentieth century to the Video Jockey 
(VJ) performances seen at the latest electronic dance music festivals, there is an 
extensive body of artistic work that seeks to visualize sound and music. The form 
that these visualizations take has been shaped significantly by the capabilities of 
available technologies; thus, we have seen a transition from paint to film; from 
hand-drawn animations to motion-graphics; and from analog to digital projection 
systems. In the twenty-first century, visualizations of music are now possible with 
extended reality (XR) technologies such as virtual reality (VR), augmented/mixed 
reality (AR/MR), and related forms of multi-projection environment such as full- 
dome. However, the successful design of visual music and VJ performances using 
XR technologies requires us to consider the compositional approaches that can be 
used by artists and designers. To investigate this area, this chapter will begin with 
an analysis of existing work that visualizes music using XR technologies. This will 
allow us to consider the spectrum of existing design approaches, and provide a com- 
mentary on the possibilities and limitations of the respective technologies. Following 
this, the chapter will provide an in-depth discussion of Weinel’s practice-led research, 
which extends from work exhibited at the Carbon Meets Silicon exhibitions held at 
Wrexham Glyndŵr University (2015, 2017), and includes AR paintings, VJ perfor- 
mances, and a VR application: Cyberdream VR. Through the discussion of these 
works, the chapter will demonstrate possible compositional principles for visualiz- 
ing music across media ranging from paint to XR, enabling the realization of work 
that reinforces the conceptual meanings associated with music. 
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12.1 Introduction 


In 2019 the Coachella music festival featured an “interactive augmented reality (AR) 
stage,” in which audiences could use a mobile application [1] to enhance the expe- 
rience through various fantastical contents, which appeared superimposed on the 
Sahara venue when viewed through a camera [2—4]. When viewed through the app, 
audiences could see “space objects like planets, asteroids, and stars” [5] within the 
performance space, providing digital visual content that complements the audience 
experience of music. This is just one example of extended reality (XR) technologies 
being brought into live performance spaces for electronic dance music. XR is an 
umbrella term that encompasses augmented/mixed-reality (AR/MR), virtual reality 
(VR), and more broadly, other associated technologies such as multi-projection 360- 
projection environments and fulldomes, all of which are increasingly being used to 
complement or visualize music at electronic dance music events, by VJs and digital 
artists. 

The use of these technologies points toward a future in which public performance 
spaces for music are enhanced through immersive XR content. Music performance 
spaces are already immersive—sounds, lighting, and communal experiences of dance 
can all contribute toward generating powerful social experiences that are emotive 
and meaningful [6]. Yet, while sound can fully engulf the audience as a spatial, aural 
experience, visual accompaniments too can go further in surrounding and immersing 
the audience. Through the use of XR, we can begin to conceive music performances 
where eventually a mesh of synthetic 3D graphics engulfs the performance space, 
creating hallucinatory computer graphics that visualize the music [7]. Yet, these 
technologies also generate a variety of new issues that must be considered through 
research. 

The main purpose of this chapter, then, is to explore how audio-visual experi- 
ences of music can be designed in XR. Specifically, I will discuss compositional 
approaches that can be used to design these experiences. I will argue for an approach 
in which designs in XR can provide visualizations of music that go beyond basic 
audio reactivity, embodying similar forms of symbolism as those present in the music, 
and enhance the audience experience by reinforcing meaningful resonances with the 
music. Towards this aim, first I will review some examples of XR music visualiza- 
tions and VJ performances, in order to evidence present activity in this field. Follow- 
ing this, I will then turn to consider my own practice-led research creating artistic 
works such as AR paintings, VJ performances, and a VR application: Cyberdream 
VR. The exploration of these artifacts, performances, and software applications will 
demonstrate compositional approaches for realizing music visualizations that cross 
boundaries from analog media such as paint, into the digital; and from digital pro- 
jections into immersive forms of XR. Through the course of this chapter, I therefore 
aim to illuminate possible approaches and open new conversations about visualizing 
music in XR. 
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12.2 The Current State of the Art 


Visual music is an established art-form with considerable history dating back over 
a hundred years to the early color organs; works by artists such as Kandinsky; and 
later the films of artists such as Len Lye, Oskar Fischinger, John Whitney, and others 
[8]. The essential priority of visual music and related forms such as the psychedelic 
light show [9] is the representation of sound and music through complementary 
visual media. From the 1980s to present, the rapid expansion and democratized 
home availability of computer and video technologies saw a significant growth in 
the related forms of music visualizations, music video, and VJ performance [10]. Of 
course, there are many possible journeys through this expansive area of work, and 
it is beyond the scope of this chapter to provide an extended history. As explored 
in a recent panel discussion that the author participated in at London South Bank 
University [11], some VJs connect their work with visual music, yet inspiration may 
also come from the wider sphere of music culture, motion graphics, film, and video. 
For our purposes here, it will be sufficient to acknowledge that music visualization 
is a rich and varied field, but also one that has been shaped significantly by new 
digital technologies over the past century. It therefore comes as no surprise that 
XR technologies are gradually being assimilated into the panoply of tools used for 
visualizing music, yet with this, they generate specific new affordances and concerns, 
which we will now discuss through a selection of examples. 


12.2.1 Multi-projection VJ Performances 


VJ performance typically involves an individual mixing live video as a complement 
to electronic dance music performances, in an approximately equivalent manner to 
the DJ [12]. In recent decades, VJ performance has developed to use multiple projec- 
tions, and video mapping techniques, which allow video to be projected on custom, 
irregular surfaces. At electronic dance music festivals such as Mo:Dem (Croatia, 
2017), elaborate sculptures provide the stage design, on to which the VJ projects 
video-mapped visuals. Besides the main stage, smaller projection screens are also 
mounted in the trees (Fig. 12.1), and complement other aspects of the festival decor 
such as ultra-violet canopies. Here the VJ projections are audio-reactive, responding 
to the beat, but the designs also reflect otherworldly, alien or shamanic symbolism, 
which complements the psychedelic themes of the music (as discussed in [13]). 
Along similar lines, Burning Man (USA, 2016) was one of many recent festivals 
to feature a fulldome theatre. Here, the fulldome provided an ad hoc movie theatre, 
which the audience could enter to view 360° films such as Samskara [14], which 
provides a cinematic experience based on the concept of a psychedelic journey anal- 
ogous to an LSD experience. The film incorporates electronic music and relates to 
the overall psychedelic theme and ethos of the festival. Elsewhere, video-mapped 
domes have also been used for VJ performances; for instance, United VJs provide 
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Fig. 12.1 Psychedelic decor with VJ projections among the treetops at Mo:Dem festival, Croatia, 
28 July 2017. Photo credit J. Weinel 


performances [15] and workshops [16] specifically addressing techniques for VJing 
in fulldomes. 

Besides fulldome, other custom multi-projection setups are often used. For 
instance, the Resolution [17] series of events in London has utilized various projec- 
tion configurations to provided extended immersive visuals. For instance, the series 
presented Sim Hutchins, whose performances revisit 1990s rave nostalgia through 
combinations of electronic music and projected visuals, on a 270° projection system 
in the G05 venue [18]. Meanwhile, another event at G05 featured music by Bobby 
Tank with a VJ performance by L’ Aubaine (Fig. 12.2, [19]), who performed on a 
360° projection system. 


12.2.2 Augmented Reality Companion Apps 


Mobile apps are often used to provide companion experiences at music events, for 
instance Notting Hill Carnival mApp [20] provides useful features such as a map of 
sound systems for Notting Hill Carnival. The Coachella app fundamentally fulfills 
a similar function, providing digital marketing and informative features; however, it 
also goes further with the incorporation of a ‘Coachella Camera’ with AR features. 
This provides AR features similar to the popular app Snapcat, allowing the user to 
take selfies and photographs with AR enhancements. However, for our purposes here, 
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Fig. 12.2 L’Aubaine performing visuals on a 6-screen projector setup at Bobby Tank Oxygen 
EP release event, by Not Like That, G05, London, 2 February 2019. Photo credit Laurie Bender 
(L’ Aubaine) 


more interesting is the capability for real-time (synchronous) experiences of location- 
specific AR content, such as the space-themed 3D imagery described in the opening 
of this chapter, which could be seen at the Sahara stage when viewed through the app. 
Here the space-theme reinforces the identity of the Coachella festival, suggesting a 
fun, exotic festival experience that seductively indicates escape from the everyday. 

Elsewhere, AR is also being used to complement or visualize music in other ways. 
The Kybalion album by Øresund Space Collective includes a mobile application 
created with artist Batuhan Bintas. The application can be activated by viewing the 
illustrations of the album artwork through a mobile device, bringing to life the artwork 
through AR/MR and VR computer graphics. Here the app allows the audience “to 
not only listen to the songs but also to learn about the Hermetic teachings of Thoth 
by interacting with the album artwork” [21]. In this regard it extends the conceptual 
universe suggested by the space rock music, effectively allowing the audience to 
enter the imaginative virtual world that the music describes. Bintas sees the work 
as revitalizing the album cover as an artifact to be enjoyed alongside the music, 
and in this regard it could be understood as complementary to the vinyl revival 
[22]. However, this piece can also be considered as a ‘cyberdelic’ (a portmanteau 
of ‘cyberculture’ and ’psychedelics’) experience, using a term that was popularized 
in the 1990s by Timothy Leary [23], and is now used by The Cyberdelics Society 
[24, 25], of which Bintas is an affiliate. Kybalion fits with the idea of cyberdelics 
because it utilizes the illusory capabilities of the technology to produce a digital art 
experience that is analogous to psychedelic hallucinations. 
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12.2.3 Music Visualizations for Mixed Reality Headsets 


Although we have noted the Coachella AR app as an example of XR aimed at the 
dancefloor, mobile phones may be poorly suited as a means of enhancing immer- 
sion in the live experience of music; indeed their use at concerts has proven con- 
tentious for audiences [26-28]. An alternative approach could be to use MR devices 
such as the Microsoft HoloLens or Magic Leap, which allow the viewer to wear 
a holographic headset that superimposes 3D content over the surrounding natural 
environment. Elsewhere, MR headsets have been used to enhance digital arts exhibi- 
tions, by bringing animated visualizations to life. For example, the recent exhibition: 
Leonardo da Vinci and Perpetual Motion: Visualising Impossible Machines [29] 
used tablet devices and a HoloLens to provide AR/MR 3D holograms, to visualize 
Leonardo da Vinci’s perpetual motion sketches as 3D computer graphics animations 
in the gallery setting. 

This area remains relatively unexplored for music visualization, but there is some 
early work, such as Synesthesia, a HoloLens app that provides an MR music visu- 
alization experience based on the generation of audio-reactive graphics [30]. It is 
possible that technologies such as these could be brought into live music events. 
However, the obvious current limitation is that the devices are too expensive to be 
widely used in the music festival environment. Significantly, the viewing angle of 
these devices is also relatively small [31], which combined with cumbersome head- 
sets would be likely to have a significant impact on immersion during a live music 
event. At the present time, MR music visualization would therefore be unsuited to the 
music festival environment, though certainly we could see visualizations of music 
in smaller scale settings such as gallery installations. As the technology improves 
and decreases in cost, it may be more feasible to organize larger scale events where 
audiences use MR glasses to view live visualizations of music. Alternatively, tech- 
nologies such as the Holo-Gauze [32] provide possibilities for projecting Pepper’s 
ghost holograms on invisible reflective materials, which can be viewed by audiences 
without headset. A similar approach using Pepper’s ghost holograms has been uti- 
lized in visual music performances by Carl Emil Carlsen for his work with Silicum 
[33], and it is possible that future technologies using approaches such as these may 
provide other ways to introduce MR into the dancefloor context. 


12.2.4 Music Visualizations in Virtual Reality 


VR is also gradually being incorporated at music festivals. For example, Psych- 
Fi [34] provides immersive experiences at music festivals such as Boomtown Fair 
and Sci-Fi London film festival. At the 2016 edition of the Liverpool International 
Festival of Psychedelia, their app Dioynsia was included in the PZYK Gallery, a 
multisensory arts installation designed to complement the festival experience [35]. 
Dioynsia provides a short journey into a hallucinatory landscape in VR, thereby 
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realizing the idea of a psychedelic trip through synthetic computer graphics and 
sound [36]. 

Elsewhere, L’ Aubaine has also created work that translates the aesthetics of her VJ 
performances into 360° narrative piece, 360 Life #1, which explores “introspection 
versus outrospection and the boundary between reality and surreality” [37]. Works 
of this type can be situated in the gallery context, but are also sometimes featured 
at VJ events; for instance, Fathomable’s Gnosis [38] is a cyberpunk VR experience 
created by VJ Rybyk, which was featured at VJ London’s AV Depot event [39]. There 
are also various other commercial VR experiences of music, which are relevant to 
consider in this section. For example, Fantasynth [40] is a short VR experience which 
provides a journey through a landscape of audio-reactive graphics, while The Wave 
VR [41, 42] and Amplify VR [43] are other music platforms that aim to provide music 
video or virtual concert experiences in VR. There are also various other bespoke 
music-related VR music experiences, such as Fabulous wonderland VR [44], a VR 
experience based on wonderland, a National Theatre musical created by Damon 
Albarn, Moira Buffini, and Rufus Norris. 


12.2.5 Discussion 


This section has outlined a selection of examples where XR technologies have been 
used to visualize music. It is significant to note that similar XR technologies underpin 
many of these productions, since 360° production software and video game engines 
such as Unity can be used to adapt XR experiences across multiple platforms. Thus, in 
some cases, where a work is designed for VR, it can be realized for other formats such 
as AR/MR or fulldome projection. The fluidity between these platforms is one reason 
why it is pragmatic for our discussion here to look at the bigger picture regarding 
the use of XR for visualizing music. Yet in considering various types of XR, we also 
find that these technologies afford different forms of audience experience. In some 
cases, multi-projection environments may be used to provide VJ experiences in the 
electronic dance music context, extending the lightshow. Yet in festival environments, 
XR technologies such as AR/MR and VR are finding new contexts, such as their use 
to provide ‘side-shows’ that audience members may experience between the main 
acts. 

It is perhaps worthwhile to consider how XR technologies may impact on dance- 
floor immersion. Dancefloor immersion is often characterized as arising from the 
experience of losing oneself in communal experiences of dance and music [45]. Yet 
Rietveld [46] argues that electronic dance music culture has undergone a shift from 
dimly lit nightclubs and warehouse parties, toward visual spectacle, which can be 
associated with the power structures of consumer capitalism. In her argument, elabo- 
rate stage designs enhance the visual and redirect the gaze of dancers away from each 
other, toward all-powerful “superstar DJs’. For Rietveld then, enhancing the visual 
spectacle of events may lead to a negative effect on collective experiences of dance- 
floor immersion. From this perspective, XR clearly carries a risk. When we consider 


216 J. Weinel 


examples such as the Coachella app, audiences are encouraged to direct their gaze 
to a mobile device, shifting activity away from the dancefloor toward social media 
interaction and the narcissistic taking of ‘selfies’. The negative impact of mobile 
phones on dancefloor immersion is already recognized in popular music press—for 
instance, a recent article in DJ Mag [47] even calls for a ban on mobile phones on 
the dancefloor, referring to academic research by Henkel [48] that suggests a ‘photo- 
taking-impairment’ effect on memory, underscoring the idea that such activities may 
reduce the presence and immersion of individuals in real-world contexts. 

However, XR may also be capable of immersing the participant in other virtual 
spaces that are distinct from the dancefloor [49]. Where VR ‘side shows’ are provided 
at festivals, these may provide virtual spaces that relate to the themes and symbolic 
meaning of the event as a whole. In St. John’s [50] discussion, festivals and raves 
provide liminal spaces of physical and social activity that are removed from the 
everyday. While the dancefloor experience may be of critical importance for these 
events, activity in these spaces is diffuse and encompasses multisensory experiences 
in which aspects such as clothing, costumes, and conversation are also significant. 
Here, XR may be complementary, since condensed experiences of digital content 
away from the dancefloor may stimulate conversation, reinforcing the meaning and 
immersion into the event as a whole. 


12.3 Case Study: Projects Visualizing Music in Extended 
Reality 


Having outlined various examples related to the visualization of Music in XR, I will 
now turn to consider my own practice-led research in this area, which includes work 
across the areas of AR painting, VJ performance and a VR application: Cyberdream 
VR. Notably, some earlier iterations of this work were presented and discussed in the 
Carbon Meets Silicon exhibitions at Wrexham GlyndWr University (2015, 2017). 
Beginning with a brief outline of earlier artistic works, in this section I will provide a 
personal journey through my creative practice, in order to demonstrate how the work 
visualizes music in different ways across a variety of media, eventually moving into 
XR territories. In doing so, I aim to illuminate some possible compositional strategies 
for visualizing music in XR. 


12.3.1 Background 


My background is in electronic music and visual arts, and was significantly developed 
through my Ph.D. [51], completed at the Keele University music studios. Here my 
work focused on the composition of electroacoustic music based on altered states 
of consciousness. In summary, this work seeks to design music that is analogous to 
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the form and structure of psychedelic hallucinations, through corresponding sonic 
materials and structures. This resulted in a series of compositions that were released 
on the Entoptic Phenomena in Audio 12” vinyl [52]; software tools that were used 
to realize these compositions; and an audiovisual composition entitled Tiny Jungle 
[53, 54]. These works organize sounds, and (for audiovisual works) visual materials, 
in order to construct experiences analogous to hallucinatory journeys [55]. 

I later extended these ideas through further audiovisual compositions: Mezcal 
Animations [56], Cenote Zaci [57] and Cenote Sagrado [58] are three fixed-media 
visual music films that seek to provide synaesthetic experiences of electronic music 
and abstract visuals, based on the concept of altered states of consciousness. These 
were widely performed at international festivals for electronic music and visual 
music, suchas the International Computer Music Conference [59], Seeing Sound [60, 
61] and others, and were included in Technoshamanic Visions from the Underworld 
[62], a loop of collected audiovisual works presented at the first Carbon Meets 
Silicon exhibition. Notably, these works use the technique of ‘direct animation’, 
where 8 mm film is hand-painted, projected and digitized, and then combined with 
other materials such as stop-motion animation and computer graphics. Around this 
time at Wrexham Glyndŵr University, I also created Quake Delirium [63, 64], a 
video game modification that seeks to represent a hallucinatory state in the form of 
an interactive video game; and Psych Dome, an interactive installation for mobile 
fulldome, in which participants wear an EEG headset that captures brainwaves, which 
are used to affect parameters of an audio-visualization based on the visual patterns 
of hallucination seen during altered states of consciousness [65]. 


12.3.2 Augmented Reality Paintings 


While working at Aalborg University in Denmark, I created several new paintings, 
which explored similar ideas to my earlier work [66]. For example, Vortex (2017) is 
based on the concept of visual patterns of hallucination, providing a funnel image 
related to Kliiver’s [67] ‘form constants’ (honeycomb, cobweb, funnel, and spiral 
forms seen during hallucinations). Alongside this work, I also began working with 
the creative coding environment Processing, designing motion graphics sketches 
related to altered states, while also drawing influence from demoscene art [68] and 
the related VJ mixes [69]. I began experimenting with mixing video live to music 
using the VJ software VDMX, combining direct animation with materials created in 
Processing and footage made using other techniques such as stop-motion animation. 
This resulted in Technoshamanic Visions from the Underworld IT [70], a pre-recorded 
video loop created by mixing video live to music by the Japanese psychedelic rock 
band Hibushibire, which was presented at the Carbon Meets Silicon II exhibition 
at Wrexham Glyndŵr University. The exhibition also featured Vortex and several of 
my other paintings, which are essentially companion pieces that test similar visual 
ideas as those I explore in the videos. 
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Continuing to explore both painting and audiovisual composition in parallel, I 
created a series of works that interpret music through synaesthetic, psychedelic forms 
of visual art. Technically these works explore the use of flow techniques, airbrushing, 
and digitally cut stencils. They incorporate other aesthetic influences from music via 
the artwork of L.A. punk bands such as Excel [71], Suicidal Tendencies [72], and 
hip-hop music via the artist Rammellzee [73]. For example Trip at the Brain (2017) 
interprets a Suicidal Tendencies song of the same name as a pen sketch, which is 
converted into a digitally cut stencil and rendered in airbrush. 37 Seconds (2017) 
incorporates airbrushed lettering, referencing a sample from the jungle track Origin 
Unknown ‘Valley of Shadows’ [74], and uses acrylic flow techniques and patterns 
that reference the designs of rave collective Spiral Tribe. Bug Powder Dust (2017) 
references the Bomb the Bass featuring Justin Warfield song “Bug Powder Dust’ [75], 
which is based on the William S. Burroughs novel The Naked Lunch [76]. For the 
latter piece, the painting uses airbrushed skeleton stencils reminiscent of Burroughs’s 
‘shotgun paintings’, to provide a form of visual quotation (or ‘sampling’, to use a 
music production metaphor) [77]. 

In many cases these paintings were created alongside the VJ work and vice versa, 
and develop similar aesthetic ideas and symbolism across these forms. I began car- 
rying out some initial experiments that integrate these practices, by video-mapping 
my VJ content on to the paintings, thereby providing visual art with moving ele- 
ments. Later, I created a series of three paintings that link the practices of painting 
and VJing by incorporating printed stills from my VJ work as collage elements: 
Enter Soundcat (2017), Soundcat S-10] (2017), and Soundcat 2000 (2017). These 
paintings were later developed through the use of an AR app, which brings the still 
images to life as VJ animations when the application is viewed through the mobile 
application, thereby reinserting the moving image elements into the paintings [78]. 
The AR paintings provide symbolic interpretations of sound and music, utilizing XR 
to link the physical media of paint with computer-generated motion graphics. 


12.3.3 VJ Performances 


My exploration of VJ performance began with improvizational jamming, in which 
video loops created with direct-animation, stop-motion animation, and computer 
graphics techniques were mixed live to various kinds of music including psychedelic 
rock and electronic dance music DJ mixes. This allowed me to experiment with 
different combinations of sounds and images. I eventually formulated this work into 
a live DJ/VJ performance under the alias Soundcat [79], which consisted of a DJ 
set utilizing various breakbeat music from the 1990s breakbeat rave era and beyond 
(e.g., [80]). In 2018 this was performed at as part of audiovisual concerts for VJ 
London at New River Studios, London (Fig. 12.3, [81]), and at a concert held at Ty 
Pawb arts centre as part of the ACM Audio Mostly in Wrexham [82]. 

The visual materials for the Soundcat performance are based on my previous 
explorations of psychedelia, while also drawing on graphics inspired by 1990s VJ 


12 Cyberdreams: Visualizing Music in Extended Reality 219 


Fig. 12.3 Soundcat DJ/VJ performance at VJ London, New River Studios, July 2018. Photo credit 
Laurie Bender (L’ Aubaine) 


mixes and demo-scene graphics [83—85]. [incorporate 3D tunnel effects and geomet- 
ric animations; 3D scenes reminiscent of the ‘cyberdelic’ science-fiction landscapes 
seen on fliers for mega-raves such as Fantazia or Dreamscape; dancing 3D figures; 
scrolling patterns referencing acid house culture through smiley faces; detourné- 
ments of the London Underground and Intel Inside logos; and other Discordian [86] 
or absurdist imagery. Branching out into the area of ‘video music’ (in which video 
samples or loops are used to create music, as exemplified by artists such as Addictive 
TV, Coldcut or Eclectic Method), one section remixes video trailers from the Planet 
of the Apes films to match the samples used in a track by Unkle (‘Ape Shall Never 
Kill Ape’ [87]). During this period I also became interested in vaporwave (Tanner 
[88]), an Internet music subculture which provides a surrealistic or hyperreal inter- 
pretation of 1990s computer graphics and techno-utopian culture, and some sections 
incorporate symbolic references to these forms through the use of computer software 
user interfaces and related symbols or designs. 

Using these materials, I created original music videos for all of the tracks that 
I wanted to include in the VJ mix. These videos were mixed live in VDMX using 
audio-reactive effects, layering, and synchronized looping techniques, all of which 
were manipulated in real time using a MIDI controller (a Korg NanoKontrol). For 
some sections, I used an Akai MPC Studio to rhythmically trigger video clips live in 
synchronization with the music by improvising with the percussion pads. For a section 
based on Equinox ‘Acid Rain V.I.P. (Breakage Final Chapter Mix)’ [89], I used the 
tracker music sequencer Renoise to program a MIDI sequence in synchronization 
with the drum track, which was then used to trigger closely synchronized 3D graphics 
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within VDMX. For each song, I created a different VJ mix, which was performed live 
in the studio, and recorded using a Blackmagic HyperDeck Shuttle. In some cases, 
further video overdubs were carried out in order to provide additional layering of 
visuals. This process resulted in a collection of original music videos for each music 
track, which could then be used to create the final DJ/VJ mixes. 

The final live performances were created using the DJ software Serato Scratch, 
the MixEmergency video plugin, an Akai AMX mixer, and the Akai MPC Studio. 
This allowed the music videos to be mixed in the same way a DJ mix would usually 
be created, where changes to pitch can be made to synchronize the beat and blend 
between music tracks. Visually, further composites were created as the tracks are 
blended, which could also be further manipulated with effects in MixEmergency 
(for example, linking EQ adjustments to color contrast). In addition, I used the per- 
cussion pads of the Akai MPC Studio to trigger ‘one shot’ audiovisual materials, 
which were layered as composites with the video mix. The resulting DJ/VJ mix 
has some limitations in that various aspects of the visuals are pre-recorded, how- 
ever, by carrying out intensive work beforehand (both in artistic and computational 
terms), the approach allows for a highly varied and efficient end result. This approach 
also prioritizes the mixing of audio and the overall structure of the DJ mix as the 
focus of effort during live performances, which is an appropriate strategy for solo 
performances where the music takes priority, and visuals are complementary. 


12.3.4 Cyberdream VR 


Cyberdream VR is a recent project that extends many of the approaches discussed 
in electronic music, painting and VJing into XR using VR. The project is based 
on the concept of a hallucinatory journey through the broken techno-utopias of 
cyberspace, providing a surrealistic world of psychedelic rave visuals and vapor- 
wave music. Cyberdream VR was created for Samsung GearVR, was adapted for a 
VR cardboard version, and has been shown at various events including Cyberdelic 
Incubator Melbourne [90], Sci-Fi London festival [91] and MIND: Past, Present + 
Future/Cyberdelics/Remote Viewing [92]. 

The experience provides a short ‘fly-through’ (approximately 5 min long), con- 
sisting of a series of scenes. The menu screen is based on the flier for the Fantazia 
NYE 1991-1992 rave event, presenting a surrealistic virtual landscape with a large 
face suspended over it. Upon entering Cyberdream VR, the viewer flies across a 
chequerboard bridge surrounded by statues of strange creatures, entering a fractal 
structure based on the Sierpinski triangle (Fig. 12.4). Next, we travel over a vast 
infinity pool with broken manikin heads floating in it. Following this, the viewer is 
suspended in a large room with airbrushed walls, which were created by digitally 
scanning paintings made with a real airbrush. In this room, an animated effect creates 
vortex patterns based on visual patterns of hallucination. In the next sequence we 
fly through a virtual sky bombarded with pop-up windows; a pastiche of the John 
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Fig. 12.4 Still image from Cyberdream VR. Image J. Weinel (2019) 


Carpenter movie They Live [93], the spam adverts are revealed as signals of capital- 
ist control. The next scene depicts a virtual chequerboard island on which Atari ST 
cursors (pixel art rendered as 3D graphics) bounce manically or lie derelict among 
Grecian statues suggestive of techno-utopianism (also a vaporwave trope, see [94, 
95]). After this, the next two rooms consist of cycling waves of brightly-colored 
cubes with oscillating color patterns and sizes. These are based on the classic ‘plas- 
mas’ of demoscene videos [96], which generate fluid animations using oscillating 
patterns—here the technique is translated into 3D, giving an impression of being 
inside the pixels of a computer monitor, while also subtly referencing the design of 
the Windows ‘95 logo. Following this, the viewer enters another airbrushed room 
(again, created using digitized hand-painted art), in which spherical objects move in 
Lissajous figures. The final scene consists of a dark, chaotic room with bouncing stro- 
boscopic arrows, and the text “the future is lost, crash the system, back to the tribes.” 
This message is a comment on the loss of the techno-utopian futures once imagined 
by cyberculture [97, 98], while calling for a dissolution and ecstatic recombination 
of these digital structures. The comment ‘back to the tribes’ also playfully hints at 
the idea of ‘technoshamanism’ (the use of technologies to access shamanic forms of 
experience, see [99]) and references free-party rave culture (e.g., Spiral Tribe). 

The soundtrack for Cyberdream VR includes short pieces of acid house, hardcore 
rave, and vaporwave music. Just as the scenes of the project are essentially artistic 
sketches, these pieces of music are audio sketches. Thematically both the visuals and 
audio are related to, and reinforce the overall concept of the piece. Rave music sug- 
gests the futuristic aspects of hardcore techno [100], while the vaporwave clips use 
a plunderphonic approach (music made from existing audio recordings, see [101]) 
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by slowing down imperfect loops of corporate library music intended to enhance 
productivity in the workplace. In this regard, the piece sonically mirrors the visuals 
through combinations of symbolic elements from psychedelic rave cyberculture and 
corporate computer culture. The overall result aims to elicit a broken, hallucinatory 
vision of the techno-utopianism of cyberculture, revealing the artificial or hyperreal- 
ity of these visions, while also hinting at the ecstatic possibility in the dissolution of 
these structures. Described another way, the piece takes Douglas Rushkoff’s [102] 
concept of the Internet as a hallucination, and attempts to visualize that hallucination 
as a synaesthetic XR experience that allows the viewer to enter into the virtual worlds 
suggested by rave culture and vaporwave music. 


12.3.5 Discussion 


The work I have outlined in this subsection spans over a decade of creative practice 
creating work related to the concept of altered states of consciousness. In different 
ways, these works represent hallucinations, and synaesthetic experiences of sound 
and music through combinations of sound and visual art. One of the distinguishing 
features of this work is that it prioritizes the visualization of music by focusing on the 
symbols and conceptual meanings that are suggested by music, rather than the phys- 
ical properties of acoustic soundwaves through audio-reactivity (though some parts 
of the work do also include audio-reactive or closely synchronized elements). In this 
regard, the work follows Danneberg’s view that music visualisers based primarily on 
audio reactivity may be relatively uninteresting, because they render only simplistic, 
readily apparent features of sound. Instead, he argues that composers should “make 
connections between deep compositional structure and images” [103]. With my own 
work, I interpret this ‘deep structure’ at a conceptual, symbolic level, where the visu- 
alization becomes a means to unlock the imaginary spatial environments and visual 
associations suggested by the music. While these symbolic visualizations can be 
realized with static visual art, animated visuals provide a way for these to be realized 
as time-based audiovisual media. XR technologies then provide a way to extend this 
idea further still, by providing spatial portals into these synaesthetic virtual worlds. 


12.4 Conclusions 


The first half of this chapter explored various examples where XR technologies are 
being used to visualize music. Through this discussion, we saw how XR provides 
new possibilities for constructing immersive visual experiences that complement 
music. These may extend the idea of the concert lightshow, or provide complemen- 
tary ‘side’ experiences that reinforce the cultural meaning of these events. The latter 
half of the chapter then discussed my own practice-led research, creating AR paint- 
ings, VJ performances, and a VR experience related to music and altered states of 
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consciousness. These works broadly seek to elicit synaesthetic experiences of sound 
and music through various forms of visualization. Through the exploration of these 
works, I have demonstrated some possible approaches for visualizing music using 
XR, and I also hope to have shown that XR technologies need not be approached as 
novelty gadgets—but rather as means through which to extend fundamental artistic 
concepts for visualizing music. The approach that I have emphasized here is one 
in which XR visualizations do not lean heavily on audio-reactivity, but rather seek 
to access deeper symbolic meanings, in order to manifest the imaginative worlds 
suggested by music as synaesthetic immersive 3D spaces. The unique potential of 
XR is to go through the portals into music that visual music paintings, films, and 
VJing have so tantalizingly provided in the past. Now, it is possible for the listener 
to enter into the music as an audiovisual space, and feel as if they are inside the 
visual worlds suggested by music, or for the visual forms of music to spill out into 
the concert hall or living room. Whether these technologies are used to visualize the 
psychedelic music, rave music, and vaporwave discussed here, or other genres, the 
potential is to radically transform the way we experience music. 
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Chapter 13 A) 
Augmenting Virtual Spaces: Affective ciecie; 
Feedback in Computer Games 


Stuart Cunningham, John Henry, and Jonathan Weinel 


Abstract Computer games can be considered a form of art insomuch as they are 
critiqued, revered and collected for their aesthetics in addition to their ludic qualities. 
Perhaps most significantly, computer games incite a plethora of emotional responses 
in their players as a deliberate and defining mechanism. However, unlike other forms 
of traditional media and art, another key feature of games is their intrinsic interactiv- 
ity, reliance upon technology and non-linearity. These traits make them particularly 
noteworthy if one wishes to consider how art forms might respond and adapt to 
their audience’s emotions. The field of affective computing has been developing for 
several decades and many of its applications have been in the analysis and mod- 
elling of emotional responses to forms of media, such as music and film. In gaming, 
recent developments have led to an increasing number of consumer-grade biofeed- 
back devices which are available on the market, some of which are explicitly sold as 
‘gaming controllers’, giving rise to greater opportunity for affective feedback to be 
incorporated. In this chapter, a review is provided of the affective gaming field. Specif- 
ically, it is proposed that these developments give rise to interesting opportunities 
whereby virtual environments can be augmented with player affective and contextual 
information. An overview is provided of affective computing fundamentals and their 
manifestation in developments relating specifically to games. The chapter consid- 
ers the impact this biometric information has upon games players, in terms of their 
experience of the game and the social connections between competitors. A number 
of associated practical and technological challenges are highlighted along with areas 
for future research and development activities. It is hoped that by exploring these 
developments in gaming that the longer established forms of art and media might be 
inspired to further embrace the possibilities offered by utilising affective feedback. 
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13.1 Introduction 


The ability to engage with an audience on an emotional level is one of the defining 
traits of many forms of art and media. The Cambridge English Dictionary defines 
art as “the making of objects, images, music, etc. that are beautiful or that express 
feelings” [1]. This supports the notion that the author of such works is able to express 
emotion, be it their own or that of others, through a variety of mechanisms. One might 
also take the view that the audience members, whilst being able to perceive the 
emotion of the author, will also have their own response to the stimulus, which may 
be consistent with the author’s emotional intent or not. Historically, this emotional 
communication has been a largely one-directional affair: the author creates a piece 
of work with its emotional intent and disseminates it in some way, the audience 
experiences the art and has their emotional response to it, and this sequence may 
repeat itself. However, there is traditionally no recourse for the emotional response 
of the audience to feedback to the author beyond the writing of reviews, critiques, 
blogs and other forms of response. 

We propose that computer games can be considered a form of contemporary art, 
much in the same way as video, film, animation and interactive audio-visual instal- 
lations are now viewed. Certainly, games combine the requisite forms of media cited 
by the dictionary definition: images and music that are visually pleasing. Games 
are also recognised for their ability to engage an audience and provoke emotional 
responses in their audience. Gee [2] advocates that the interpretation of their aesthet- 
ics and the story aspects of games qualify them as art. Further, we note that games 
are critiqued, reviewed, discussed and are points of cultural reference. Smuts [3] 
shares this perspective, although notably recognises that not every computer game 
may qualify for the label as being recognised as art. One characteristic that makes 
games particularly compelling as an art form is the rate of change and development 
with which the media evolves and its facility to integrate easily with other techno- 
logical developments, both hardware and software based. Therefore, as a form of 
interactive art, games are uniquely poised to trailblaze and be the proving ground of 
new ideas and initiative, which may later be adopted by the wider artistic community 
in all forms of practice. Melissinos [4] states that “... games are also the only form 
of media that allows for personalizing the artistic experience while still retaining 
the authority of the artist,” making the distinction that narrative and content of the 
work can be tailored to the individual(s) playing. It is on these bases that we consider 
games as being a platform to develop work that affords interactivity, in terms of 
audience affect and immersion within the game. Further, we propose that this can 
lead to develop and enhance a sense of community and connection between human 
players in a game environment, who are remote to another. 
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Many years have now passed since Rosalind Picard produced the seminal work 
that helped to define the field of Affective Computing and its associated challenges, 
opportunities and potential risks [5]. Key to the domain are the three types of affective 
technology Picard outlines where computers are able to: recognise; represent and 
have emotions. Whilst the former of these is of greatest concern in this chapter, the 
second element of representation of emotion should not be neglected. The field itself 
has developed, at least on the front of academic and some industrial research, over this 
time period. However, its mainstream applications in the hands of the audience are 
still relatively few and far between with exceptions such as the formation of spin-out 
company Affectiva' from Picard’s MIT Media Lab. Perhaps this is unsurprising, as we 
have seen the widespread adoption of other paradigm-shifting art and entertainment 
technologies, such as virtual reality (VR), take a long time to reach the mainstream 
and gain acceptance from publishers and audiences alike. 

A common underpinning element of affective computing is the reliance upon 
various sensors and mechanisms for the provision of biofeedback; the mechanisms 
by which aspects of bodily function and response can be identified and manipulated 
[6]. Whilst not an essential aspect of all affective computing systems, the use of 
such technologies is an intuitive method that provides indicators of affective state 
and responses to stimuli, often with the ability to operate in real-time. There are 
distinct advantages of using such sensors. For example, subjects, in our case games 
players, do not need to actively focus on providing feedback as they would with self- 
report mechanisms, rather data can be collected passively and without distracting 
attention from the medium itself. Additionally, sensors are able to provide objective 
measurements, whilst asking for self-reports or other information may introduce 
bias on the part of the subject. Whilst medical grade equipment is desirable in high- 
resolution research scenarios, at a consumer level there are a range of technologies, 
some of which can provide viable affective data [7—10] and be easily integrated or 
added to gaming devices and apparatus. 

We propose that affective gaming is a natural extension of the affective computing 
field. Such an extension offers game developers, the opportunity to detect, respond 
to and provoke emotions in players. Computer games, like many forms of art, media 
and entertainment, offer their players the opportunity to be emotionally stimulated 
and involved [11, 12]. One of the key features that separates games from other 
forms of media-based entertainment is its broad non-linearity and its deep and varied 
levels of interactivity. As such, affective gaming affords the possibility to create 
gaming experiences that can not only provoke and influence their players’ emotional 
responses, but that player emotions can also be used as an input to the game itself, 
allowing it to respond to the player and provide a customised experience. This process 
is generally referred to as affective feedback [13]. One clear example would be a game 
that increases its level of difficulty to increase tension in the player, if it were to detect 
that the current level of difficulty is not sparking the intended response. 

The ways in which end users interact with technology is maturing and tradi- 
tional mechanisms of button pressing, joystick moving and mouse clicking have been 
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ubiquitous and effective. However, these technologies have become a legacy, partly 
down to reluctance in the market to engage with alternate methods of input, such 
as motion detection, speech and other modes of interaction. We suggest that, in the 
coming years, the time will be right to examine both active and passive methods 
of interaction with games and other digital media. There are multiple possibilities 
to create experiences that are powerful, intelligent, customisable, adaptive, socially 
enabling and responsive to their users. These developments represent exciting oppor- 
tunities to create innovative entertainment and artistic works that enhance and evolve 
the user experience. 


13.2 Affect, Immersion and Engagement 


In Inner Sound: Altered States of Consciousness in Electronic Music and Audio- 
Visual Media [14], Weinel proposes a conceptual model that considers the ‘affective 
properties’ of sound and audio-visual media as a central feature. Here affective 
properties are considered as related to the properties of valence and arousal described 
in Russell’s [15] circumplex model of affect. In games, sound and music may help to 
elicit an emotional response from the player. Following Gabriellson and Lindstrom’s 
[16] meta-study of music and emotion, typical features such as percussive sections 
of music may be associated with high arousal, while sparser rhythmic sections may 
relate to low arousal. Major modes may relate to positive valence, while minor modes 
may suggest negative valence. While these are general correspondences that may not 
apply in all cases, such typical associations can still be used by games designers to 
reinforce the affective experience of games. We commonly see this through action 
sequences that are scored with up-tempo music, or failure or melancholy sections 
in game narratives that are reinforced with corresponding music. For Weinel, such 
features help to provide an affective experience that is framed by representational 
properties, and the overall experience provides a heightened state of multimodal 
awareness that may encompass forms of experiential knowledge. 

Of course, in the gaming context, affective states do not only relate to music. 
Cajella’s [17] work on ‘player involvement’ breaks down immersion in video games 
into the categories of kinaesthetic, spatial, shared, narrative, affective and ludic forms 
of player involvement. While this model takes the affective experience as a distinct 
component, it is surely a form of involvement that must emerge from all of the 
others. For instance, kinaesthetic involvement describes engagement through control 
of a game character, while spatial involvement relates to the virtual environments 
provided by games. When a player controls an avatar that leaps through the levels of 
a game, they are engaging kinaesthetically and spatially, but the feeling of excitement 
that may develop from this also leads to affective involvement. Similarly, for players 
who enjoy the shared involvement of multiplayer combat games, an added sense of 
arousal comes from knowing that their own success or failure has a bearing on the 
emotions of their human competitors. For games that develop complex narratives, as 
with films, designers may aim to tell a story that has an emotional resonance with the 
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player. Lastly, ludic involvement also generates arousal, as players attempt to evade 
losing lives or reaching a ‘game over’ state, which inevitably is disappointing. Hence, 
we can see that in fact, all types of involvement feed into the affective involvement 
of the player. Sound and music are by no means the only way through which affect 
can be generated in interactive games, but these are often used to help reinforce and 
heighten the affective experience of the player. 


13.3 Social Presence and Connection 


Weinel’s [14] work on altered states of consciousness also draws some further con- 
nections between the affective properties of media and social or shared experiences. 
This emerges through Rouget’s [18] seminal work on music and trance in various 
ritual contexts such as those found in shamanic cultures. Rouget draws a distinction 
between high-energy states such as trance, which could be understood as extreme 
states of high arousal, and meditative states of contemplation (referred to in the orig- 
inal text as ‘ecstasy’), which conversely may be thought of as low arousal in form. 
Rouget explores a variety of ritual contexts in which these states may be used in 
cultures across the globe, and comments on how music can be involved in structur- 
ing and producing these states. Yet interesting, high-energy states such as trance are 
also distinguished as social, involving loud noises, movement and other techniques 
of sensory overload such as the spraying of liquids; while meditative states are char- 
acterised as occurring in darkness, solitude and related to a reduction, narrowing or 
focusing of the senses. 

There has been little work done on considering video games as altered states of 
consciousness, but clearly this is a popular form of media that perhaps is valued by 
audiences for its capability to induce a sense of (secular) trance. Is it not possible that 
as teenagers play popular games such as Fornite [19], the heightened state of excite- 
ment they experience, which draws them out of the everyday and into the virtual, 
is in fact something we might call a state of trance? Various interactive aspects of 
the media conspire to produce a highly aroused, social experience, in which they are 
immersed into imaginative virtual worlds of play. In these terms, it seems that there 
is no shortage of high-arousal action games that aim to produce something we might 
call a sense of trance. But what then, of more tranquil or meditative states? In fact, 
there are also various games that specifically aim to induce states of meditation, such 
as Guided Meditation VR [20]. This title is essentially the VR equivalent of a guided 
meditation experience, which provides tranquil virtual locations with an instruc- 
tive audio narrative. More broadly, other single-player games may seek to provide 
tranquil, solitary experiences. For instance, ABZU [21] is a mysterious underwater 
exploration game that de-emphasises ludic involvement in favour of kinaesthetic, 
spatial and narrative involvement, as the player navigates mostly peaceful ocean 
environments with a relaxing soundtrack. Certainly, games such as this could be 
understood as offering experiences that are characterised as meditative in form. 
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13.4 Augmenting Virtual Environments 


The field of Augmented Reality (AR) has developed significantly in recent years. 
From the perspective of a user or consumer, it seeks to combine information from the 
real world with information not visible in that environment using the standard array 
of human senses [22]. Potentially, this additional information may be drawn from 
imperceptible information that exists in that space or by including extra information 
from another place: real or virtual. Thus far, the majority of AR systems have sought 
to overlay a user’s everyday environment with additional information. For example, 
an AR navigation application might overlay information about a car that is being 
looked at, such as year of manufacture, engine size and so on. Alternately, a real 
worldview might be overlaid with the user’s schedule for the day. There have been 
several commercial attempts at developing AR systems of this kind, perhaps most 
notably the Google Glass” hardware project, as well as an array of smartphone apps 
developed commercially or as research projects. 

However, less common is the ability to apply the same principles to augment a 
Virtual Reality (VR) space with information from the real world. For example, a 
virtual tour of a building to be constructed might be presented by replicating the 
current weather being experienced in the real world by the user. Notable consumer 
devices that explore these possibilities are Sony’s PlayStation Camera [23] and 
Microsoft’s Kinect for Xbox One [24]. These platforms have seen video games being 
developed that insert, or respond to, a live camera feed image of the player in a game 
environment, giving them the ability to interact by gesture with the game, hence 
augmenting the VR environment. This builds upon the more established practice 
of integrating audio, specifically player speech utterances, into game environments, 
made popular in online multiplayer gaming. It is this latter variation of augmentation 
that exemplifies some of our concerns in this chapter. 

Our motivation particularly stems from the recognition that playing games in 
virtual or online environments has many differences from their real-world equiva- 
lents. For instance, there is a reduced ability to interact with fellow team members 
and opponents. For example, there is a distinct absence of interaction through facial 
expressions, gesture, posture and general body language. Even with the presence of 
speech utterances and emoticons, it is challenging to determine accurate emotional 
responses and exert forms of emotional intelligence, to determine if other players 
may be angry, sad, scared, exhilarated and so on. As recognised by Picard [5] in 
relation to other forms of electronic communication, such as email, this detracts 
from the entertainment and immersion within the game and also from the ability for 
players to successfully interact with one another. To address this, we propose that 
these differences lead to opportunities to improve players’ sense of immersion within 
a virtual environment and their social experience with other players in that environ- 
ment. However, the drive for increased immersion and realism in games and virtual 
worlds leads to causes for concern regarding the health issues associated [25] and 


*https://developers.google.com/glass/. 


13 Augmenting Virtual Spaces: Affective Feedback in Computer Games 235 


thus any experimental approach must be mindful of the ethical issues surrounding 
such work. 

The use of standard biofeedback is fairly commonplace, whereby real-time repre- 
sentations of the underlying signals in question are presented to a person connected 
to some form of sensor. Perhaps the most familiar manifestation of this would be 
an electrocardiogram (ECG) machine in a clinical environment, where the person 
connected to the device would be able to hear and see their heart rate. Similar body 
sensors allow games to alter their virtual environments based on the user that is 
playing them. Elaborating, emotion is quantifiable through sensor measurements. A 
virtual game environment becomes dynamic, if it reacts to emotion. A strong case 
study for such an application is Dynamic Difficulty Adjustment (DDA) in games. 

A game such as those from Electronic Arts’ FIFA [26] series can generate an 
enormous amount of frustration for players; with a number of players reporting they 
quit playing the game because they felt it unfairly caused them to lose. In future 
consoles, controllers could embed heart rate monitoring sensors. The data produced 
could in turn generate a crude measure of emotion, calculated through computer 
algorithms that translate raw sensor data to meaningful information. Therefore, the 
level of emotional distress becomes the value for adjusting the difficulty in the game. 
This adheres to the theory of flow by Csikszentmihalyi [27]. The channel of flow 
exists between challenge and achievement. A game that dynamically alters its dif- 
ficulty based on the player’s emotion, can help keep gameplay within the channel. 
The Internet of Things (IoT) and serious games provide more detail and background 
on this topic. 

The ability for the affective information about players to be shared with third- 
parties [28], such as in online multiplayer gaming environments, presents interesting 
opportunities. Consider the illustrative example of a mock-up interface provided in 
Fig. 13.1, where the virtual game environment has been augmented with data about 
the human players controlling the in-game avatars. In this simplistic example, we 
observe the presentation of raw biosensor data alongside a higher-level summary of 
affective state. In this exemplar, raw heart-rate data in beats-per-minute is presented 
numerically in a heart icon on the left of the pop-out window. In the middle is an emoji 
icon, which might be derived by a range of different sensors or other mechanisms, but 
with the intention of providing a summary representation of the player’s emotional 
state. Finally, on the right side of the pop-out a lightning bolt, which fills up with 
an orange colour, provides an indication of the galvanic skin response (GSR) data 
values of the player. Although visual paradigms and designs have been adopted for 
the purposes of communicating these concepts clearly in a written form, there is of 
course no reason why this information could not be communicated in alternate mode, 
such as sonification or tactile feedback. 
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Fig. 13.1 Multiplayer game mock-up with affective data visualisations. Image Copyright © Stuart 
Cunningham 2019 


13.5 The Internet of Things and Games 


Research is helping bridge the gap between sensors and games. In 2015, Favorskaya 
et al. [29] presented their book into smart serious games. The idea behind the term is 
games that incorporate the benefits of the Internet of Things (IoT) with the applica- 
tions of serious games. Existing research in the field investigates measuring student 
engagement through the IoT and serious games [30]. Developing data algorithms 
forms a key component for the integration of IoT with games. It is possible to exclude 
them in games where sensors are limited to direct input. An example of such games 
is Kinect Sports [31], where the Kinect acts as an input device for controlling a player 
in the virtual environment. However, such games constrict their scope of application, 
as the data could be interpolated into something grander and more meaningful. 

However, this idea can become more abstract, and potentially, more beneficial 
if we consider the combination of IoT with games technology. Henry et al. [32] 
produced a framework that illustrates the backbone for merging serious games with 
IoT, however as the framework did not focus on a case study, it is also valid for the 
broader spectrum of game technologies. The potential for this line of research is near 
unlimited. 

Figure 13.2 illustrates the untapped potential for this academic domain in 
the present day. This validates the theories and ideologies presented in this 
chapter, and produces a stepping-stone to realising them. Research is continuing 
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Fig. 13.2 Potential applications for Henry et al. framework on serious games and the Internet of 
Things 


into understanding the relationship between all aspects of game applications and 
harnessing their potential. 

Research is emerging into combining IoT with Gamification applications. Crow- 
ley et al. [33] proposed a system where users can report issues within their local 
environment through gamification. Users would be awarded for contributing and 
could send the information through their smartphones. Research by Islam et al. [34] 
into advancements of IoT healthcare applications suggested wearables to bridge 
patient engagement and public health information. In industry, smartwatches have 
turned this concept into reality. Companies such as Apple utilise sensors to monitor 
the activity of its user and promote healthier habits by awarding badges and forming 
competitions with friends that own the same product. This is an example of gam- 
ification and IoT in applications. The method for engaging users with their health 
and maintaining their commitment is gamification, utilising game elements to an 
application outside the definition of a game. 

It is evident that games are much more than just entertainment software, and 
this remains true for game technology. Research utilises game technology to solve 
specific problems in domains such as business modelling and smart cities. Maines 
et al. [35] introduced a third dimension for Business Process Model and Notation, 
in an attempt to overlay the security requirements of a system in its design stage, 
therefore increasing its security at implementation. This solution was developed 
utilising the Unity 3D game engine. If Maine’s research were to introduce IoT, it 
could assess existing physical and virtual systems and highlight vulnerabilities based 
on current standards. Tully et al. [36], presented a methodology for rendering large 
map data for crisis management. This application ties into the principles of smart 
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cities and provides perspective to the future tools that emergency services may use 
to run more efficiently during a crisis. 

When reflecting on the progress research is making in this domain, it becomes 
evident that sensors and the IoT generate more than novel applications or theoretical 
frameworks, but a new perspective. This perspective allows all existing research into 
serious and traditional games to be re-examined with the inclusion of the IoT. The 
following section describes the technical challenges developers of “smart serious 
games’ solutions face today, and how they may be tackled in future. 

Regarding the combination of IoT and serious games, the biggest technical chal- 
lenge is heterogeneity. As the number of manufacturers producing Internet intercon- 
nected solutions increases so does the competition, driving prices down and encour- 
aging exploration. However, this competition generates segregation in the industry 
and causes development to be cumbersome. There are devices that only work within 
a manufacturer’s ecosystem, such as Samsung’s ecosystem of Smart appliances, 
devices that require bespoke bridges to connect with the Internet, and devices with 
bespoke online portals. 

The only method for tackling the increasing segregation in the Internet of Things 
is standardisation. Wireless charging is an example for the effective use of standardi- 
sation. Without the Qi standard [37], devices would require their own type of wireless 
charger. In such a scenario, large technology companies such as Apple and Samsung 
would develop their own versions of wireless charging that would only be compatible 
with their ecosystems. Therefore, wireless charging in cars or in embedded surfaces 
would become challenging. 


13.6 Measuring Affective Response 


Sykes and Brown [28] conducted a small-scale investigation to determine how the 
affective response of players would be influenced by increasing game difficulty in 
a remake of the classic arcade game Space Invaders [38]. Their study employed a 
sample of ten participants who each played the game at three randomised levels of 
difficulty. During each period of gameplay, the pressure with which players pressed 
the analogue button to move their game character was recorded. The authors argue 
that the pressure of button presses may give an indication of the player’s level of 
emotional arousal, drawing upon earlier findings in the field of sentics [39]. The 
results obtained gave an indication that at the hardest difficulty level, the pressure 
applied was significantly different than at the easy and medium levels. As an early 
work these results are compelling, although a larger sample would be desirable along 
with more rigorous statistical analysis of the data. There is doubt about the ability 
of touch universal method to communicate human emotion [40], although for most 
games players this may be an intuitive and appropriate scenario and might be one 
that could be learned for each individual over a period of time. Nevertheless, the 
method employed is interesting, especially since the authors justify the use of button 
pressure measurement as being something that can be employed in game controllers 


13 Augmenting Virtual Spaces: Affective Feedback in Computer Games 239 


with analogue buttons, meaning that more elaborate equipment and setups can be 
avoided. This is hugely beneficial given that the control devices, such as gamepads, 
touchscreens, buttons and joysticks, are the main physical interaction mechanisms 
that the player has with a game. 

Following the convention of making use of game controllers for affective feed- 
back, Bacchini et al. [41] attempted to integrate several biofeedback mechanisms 
into a common gamepad design. As such, they augmented an existing console style 
control pad with sensors designed to detect GSR, skin temperature, pressure/force 
and heart rate statistics via photo plethysmography (PPG). Thus, the device itself is 
one that is already well accepted in the gaming community and presents minimal 
disruption to the ergonomics of the typical gamepad. Their work was undertaken with 
the specific design intention that such a device would lend itself to affective gaming 
applications, which might be used for a variety of purposes, including entertainment 
as well as serious game situations. The gamepad was evaluated using a linear hor- 
ror game named Death: Unknown [42], which was an independent game available 
freely to download. The device itself showed promise in being able to successfully 
detect scare events in the game. Data obtained in the reported trial of 31 participants 
shows promise in being able to successfully detect expected scare events. However, 
the analysis makes limited to no use of additional sensor data and is largely focused 
upon use of GSR data. 

Simões et al. [43] developed a serious game to help individuals with Autistic 
Spectrum Disorder, measuring the levels of independence and performance in the 
context of a serious game designed to simulate the performance of an everyday task— 
planning a bus route, making the journey and interacting with others along the way. 
The key metric to measure anxiety in this scenario was via the use of electrodermal 
activity (EDA). It is interesting to note that the VR game itself attempts to mimic real 
life (to a reasonable degree), rather than obtusely stimulate emotional responses. 

Kivikangas and Ravaja [44] report upon the affective responses encountered in 
users playing against one another in a multiplayer scenario compared with playing 
against the computer. Measurements of emotion were recorded using electromyo- 
graphy (EMG) and EDA sensors. As such, there was no direct affective feedback 
loop returned to the game nor was there specific measurement of pre-defined game 
mechanics designed to provoke emotion. Rather, the authors were evaluating the 
effect of the outcome of playing the game (winning or losing) and how the relation- 
ship between each player and their opponent (whether they were friends, strangers or 
a computer-controlled character) impacted their emotions. Their study involved 33 
participants and used the game Duke Nukem Advance [45], a version of the classic 
first-person shooter game adapted to be played on a portable gaming console; the 
GameBoy Advance. The findings showed that positive affect occurred every time the 
player won a game, but also when they lost. The positive responses to victory became 
stronger when playing against another human, rather than a computer-controlled 
character. One intriguing anomaly is that the authors note “...a curious effect of 
negative response to victory over a friend, for which we presented numerous pos- 
sible explanations, although we cannot say which one would be the most plausible 
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Fig. 13.3 Game controller equipped with bio-sensors (points 39a and 39b) in Sony Patent [46] 


one.” Broadly, the work is supportive of the concept that relationship is an important 
factor in the intensity of emotional response in players. 

An early study of how affective feedback can be used to influence in-game actions 
used a simple competitive two player racing game, appropriately entitled Relax-to- 
Win [13]. The notion of incorporating biofeedback has also received attention from 
a number of large games console manufacturers, with a notable example being the 
patent held by Sony [46] that depicts various interface and game control devices 
that have notably been fitted with bio-signals, shown in positions 39a and 39b in 
Fig. 13.3, designed to be used as inputs to the game. The patent goes on to discuss a 
range of such signals and systems that may be used to generate data to be sent to the 
game or interactive software. Such an approach is flexible and avoids over-reliance 
upon a particular type of signal. The validity of skin response equipment may be 
brought into question under certain gaming circumstances, especially in demanding 
scenarios where intense movement of the muscles, usually in the hand, is required 
[28]. 


13.7 Suggestions for Future Work 


13.7.1 Measuring the Efficacy of Affective Games 


An initial piece of research to be done would examine the effects of incorporat- 
ing biofeedback within networked multiplayer video games. This research could 
use a participatory study to demonstrate the effects of biofeedback technologies on 
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the experience players have of the video game, particularly with regard to their immer- 
sion, sociability, experience and overall enjoyment of the game. This would make use 
of a range of bio-sensors attached to players of the game. For the purposes of a start 
point, we suggest that indicators of players’ ECG, EDA and electroencephalography 
(EEG) are selected. 

The research could provide a simulated game environment that displays biofeed- 
back information about competitors on the computer screen, much like the mock-up 
presented in Fig. 13.1. Suitable methods of visualisation and sonification of this 
data may be used, which would enable participants to perceive the data in an acces- 
sible manner appropriate to the video game medium. Participants would play the 
video-game in a head-to-head or collaborative scenario, comparing the video game 
experience without biofeedback to the one with biofeedback (one group of partici- 
pants will play with biofeedback, another group will play without). As such the study 
would broadly follow the approach of a randomised controlled trial. Both subjective, 
self-report data, using instruments such as the GEQ outlined earlier could be used to 
capture player experience alongside other objective measures such as playing time, 
success metrics in the game itself, video footage of each player and so forth. 


13.7.2 Affective Games and Altered States of Consciousness 


In previous work, several prototypes were created which explored the use of 
consumer-grade biofeedback headsets as controllers for interactive artworks and 
games. One of these: Psych Dome [47] utilised a NeuroSky MindWave to control 
the generation of a visualisation and sound in a mobile full-dome. While the design 
of the sound and visualisation were based on the concept of altered states of con- 
sciousness, using biofeedback allowed properties of the simulation to be linked to 
the brain activity of the person seeing it. This type of interaction is largely non- 
volitional (that is, the individual tends not to be able to control their brain in such 
a way that they can choose to influence the signals in a particular way), but instead 
provides a form of ‘passive’ interactivity that can enrich the sense of interactivity. 
In a related project, Quake Delirium EEG [48], this passive interaction was again 
provided for an adapted video game, which aimed to simulate an intoxicated state 
of hallucination, for which graphical and sound properties were linked to the EEG 
signals. These projects are based on practice-led research, in which prototypes are 
developed, but the prototypes can then use empirical research approaches to measure 
the experience of participants. Eventually such approaches could lead towards the 
possibility of ‘altered states of consciousness simulations’, as proposed by Weinel 
[14]. 

In relation to the latter, it has been shown that there are significant therapeutic 
benefits that can arise from altered states of consciousness facilitated by drugs such 
as MDMA, LSD or psilocybin (e.g. Mithoefer et al. [49]; Bogenschutz and Johnson 
[50]). Technologies such as VR have been widely touted as capable of simulating 
real-world experiences that could have therapeutic applications—could VR also be 
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used to effectively simulate an altered state? Some early projects such as Guided 
Meditation VR [20] hint at this possibility, yet there are also good reasons to be 
sceptical about the benefits such experiences can actually provide when compared 
with real meditation practices. What is therefore needed is research that takes bespoke 
designs aimed to simulate altered states of consciousness through games or VR, and 
measures the response of players, in order to demonstrate the extent to which these 
designs are effective. Such research can then feed back into the generation of new 
designs, perhaps allowing us to create virtual experiences that are tailored towards 
particular kinds of perceptual experiences, eventually simulating altered states of 
consciousness. 


13.7.3 Measuring the Social Aspects of Multiplayer Affective 
Games 


Research on the level of connection and social presence [51] between players in 
multiplayer gaming scenarios is underrepresented when compared to the amount 
of information available about the single player experience. This is especially true 
when considering the range of sensor data that affective gaming systems can provide. 
Ekman et al. [52] recognised this, defining the metric entitled physiological linkage 
and postulated that it may be associated with the social presence experienced by 
players. The concept itself is simple to understand: the physiological measures of 
players are likely to display patterns of synchronicity or correlation to one another, 
meaning that it “...can be used as a measure of the intensity of the interaction between 
participants” [52]. 

As explained earlier, when dealing with affect in multiplayer games it would 
be highly desirable to examine how social elements of the gaming experience con- 
tribute to player affect and, in turn, how the feedback of a player’s affective data might 
influence other players. Accurately determining multifaceted concepts of game user 
experience such as social presence, immersion and flow can be tough when using 
only biosensors. Similarly, mechanisms to verify the affective responses predicted 
from these sensors are also desirable. As such, a challenge exists in being able to mea- 
sure these complex phenomena of the human condition. It is here where self-report 
mechanisms become useful and there are several good examples of tools to measure 
many of these concepts. In their work, which also highlighted objective measures for 
immersion, such as eye-tracking, Jennett et al. [53] developed a series of self-report 
questionnaires to be issued to games players. Factors they recognised as signifying 
immersion in players included: affect; cognitive involvement and detachment from 
the real-world. Contributing to the level of immersion experience were also factors 
produced by the game itself, namely, that of challenge and control. Whilst Jennett 
et al.’s work deals with measuring the experience of the individual, it was not pur- 
posely intended to examine social aspects and connection with other human players 
in the game. 


13 Augmenting Virtual Spaces: Affective Feedback in Computer Games 243 


One of the most commonly employed tools in the field of computer games research 
is the Game Experience Questionnaire (GEQ) and its component instrument, the 
In-game GEQ [54]. The tool integrates aspects of: competence; immersion; flow; 
tension; challenge and affect. Additionally, it offers a social presence module, which 
is designed to “...investigates psychological and behavioural involvement of the 
player with other social entities...” [54], where these social entities may be other 
human players or computer-controlled characters. The social presence element of 
the GEQ consists of a total of seventeen questions that participants rate using a five- 
point Likert scale. These questions provide scores under three themes of: empathy; 
negative feelings and behavioural involvement. The first two of these have a strong 
resonance in affective gaming, leading to useful indicators of how well an affective 
game might be communicating the emotional state of other players (empathy) and 
how this information might impact the player observing such information (nega- 
tive feelings—incorporates elements of influence from others as well as a sense of 
revenge or retribution). Lastly, behavioural interaction, seeks to determine how the 
in-game actions of other players would change what the player in question did in the 
game, as well as providing some indication of how meticulously players observed 
one another. Consequently, the GEQ appears to be an effective mechanism for the 
subjective evaluation of experience in affective multiplayer games. Its social presence 
module certainly takes a step in the right direction to gauge the nature of interaction 
and influence between players, but it would be useful to consider other instruments 
that attempt to measure how connected or ‘close’ players feel to one another dur- 
ing gameplay. A solution to this may be to append the In-Game Relatedness items 
from the Player Experience of Need Satisfaction (PENS) questionnaire [55]. Such a 
hybrid approach, combining overlapping aspects from a range of game experience 
instruments is advocated by the work of Denisova, Norden and Cairns [56]. 


13.8 Ethical Considerations 


There are ethical concerns relating to the combination of serious games and IoT. 
Games in traditional forms store little information regarding their players—com- 
monly scores, an alias or their real name. The advances in mobile gaming have 
transformed the data that a game can access. Numerous games require you to sign 
into a social media account. Once you have signed-in, the company has access to 
personal information about the player and a list of the player’s friends. Introducing 
the IoT only increases the amount of information games would retain about its users. 

As discussed, an IoT enabled game could record a player’s heart rate to dynami- 
cally alter a virtual environment, or help a player progress in it. The idea of a game 
company holding such personal data could become uncomfortable to players; how- 
ever, there are arguments to counter this perspective too. As younger generations are 
introduced to technology from an increasingly early age, their perception of shar- 
ing data could be different in comparison to generations that were introduced to 
technology at an older age. 
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Irrespective of the discomfort users may have, there is an irrefutable fact that 
games of this nature will need to store very sensitive information on its users. This 
raises concerns regarding security. The General Data Protection Regulation (GDPR) 
law in Europe [57] is a conscious effort to increase liability on developers regarding 
storing personal data securely. This pressure increases the priority of securing new 
interconnected systems and the data that flows within it. However, there are still 
many challenges that the Internet of Things faces regarding security, many of which 
stem from the lack of standardisation. Therefore, when embedding IoT with Games, 
these challenges will relate to games too. 

Associated with the work discussed here, which proposes to utilise affective data 
and bio-sensor signals as input to game environments that may be socially oriented, 
are concerns over psychological effects. Consider the first research activity proposed 
earlier, for example, which would involve the visualisation and sharing of bio-sensor 
information in collaborative or competitive games between human players. Certain 
parallels might be drawn with aspects of the Stanley Milgram’s controversial experi- 
ment [58] into behaviour and obedience. If one player of a game is competing against 
another and sees that their opponent’s heart rate has elevated to abnormally high- 
levels, will they attribute this situation to themselves and would guilt and blame be 
attributed to them should their real-life opponent suffer cardiac problems? Perhaps 
the answer is that the sensor attached to their opponent is simply faulty, but without 
knowing this, the player might still suffer anxiety or low mood as a consequence. 
One rebuttal to these fears may be that multiplayer gaming is not a new concept and 
so simply presenting this information is only making visible a phenomenon that may 
have been occurring already. Nevertheless, there is a duty of care to be considered 
once this information has come to light. As recently pointed out in an article by 
Tidsdale, “...video gaming has been claiming victims for decades” [59] and with the 
enhanced sense of immersion, social connection and presence that we propose here, 
we would be wise to approach research in this field with due care and diligence. 
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Abstract Chandini (A Bride For The Moon) is an art/science project led by R&D 
Studio which explores the dream of India’s ambition to be the fourth nation to make 
a soft landing on the moon. In collaboration with a dancer, musician and scientists, 
poetic visual and performative metaphors are developed that represent the progress 
of Indian society in the technological age. This chapter describes the ongoing project 
from the authors’ perspective; how they collaborated, in response to different oppor- 
tunities and changing circumstances, the obstacles they encountered and how the 
public engaged with their artworks at events and festivals between 2017 and 2019 in 
the UK and India. 


Keywords Filmmaking - Choreography - Art/science - Collaboration - 
Documentary - Projection mapping + Outdoor arts - Space - Lunar landing - 
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14.1 Introduction 


Chandini (A Bride For The Moon) is an art/science project led by R&D Studio [1] 
that explores the hopes and dreams of India’s ambition to be the fourth nation to make 
a soft landing on the moon. Working in collaboration with artists and scientists they 
develop poetic visual metaphors to consider notions of progress to Indian society in 
the technological age. 

R&D Studio is a coming together of two artist filmmakers; Rachel Davies and 
Daniel Saul. They describe here how their artistic responses to a subject have evolved 
creating various iterations of the project over a two-year period. 

Both artists have experience of working with Indian documentary subjects and 
dancers. Rachel with short Channel 4 dance films, collaborations with Mavin Khoo 


R. Davies (EX) 
Kingston University, Kingston, UK 
e-mail: r.davies @kingston.ac.uk; rachel @racheldavies.co 


D. Saul 
Royal College of Art, London, UK 
e-mail: d.saul @arts.ac.uk; dan @rachelanddaniel.co 


© The Author(s) 2020 251 
R. Earnshaw et al. (eds.), Technology, Design and the Arts—Opportunities and Challenges, 
Springer Series on Cultural Computing, https://doi.org/10.1007/978-3-030-42097-0_14 


252 R. Davies and D. Saul 


(Khooyile) [2] and Akram Khan (Loose in Flight) [3] which toured internationally 
with the British Council [4]; and Daniel’s film The Morris Jelly House of Fashion 
(Channel 4) [5] which drew on his personal experience of being from an Anglo-Indian 
family originating from Calcutta. 

The artists remember when India was characterised in the UK as an indigent recip- 
ient of foreign aid. However recently India has positioned itself on the global stage 
with a symbolic ascendency to become a self-proclaimed leader in space technology. 
This project reflects on these contrasting contemporary global perceptions of an India 
which has been transformed by technology yet in many ways whose traditions have 
stayed the same. 

For Chandini (A Bride For The Moon) R&D Studio worked with Hemabharathy 
Palani a dancer/choreographer [6] and TeamIndus [7] a private aerospace company, 
both based in Bangalore, to produce short films, choreography, performances, pro- 
jection mapping and installations that engaged audiences through broadcast and 
performances in the UK and India. 

The wider project began as a response to an open call from the British Coun- 
cil/Big Dance Shorts in 2017 [8]. The commissions were seeking dance and film 
collaborations between the UK and India to mark 70 years of Indian Independence 
from Britain, a complex relationship from the outset. It was then that R&D Studio 
decided to focus on a small private company of mainly young people who were late 
entrants to the Google’s 2007 Lunar X-Prize; a $30 million international competition 
open to non-governmental agencies to land a robot on the Moon, have it travel 500 
m, and broadcast high resolution captured images back to Earth [9]. 

TeamIndus [10] presented themselves in a markedly different way from more 
grandiose national Space agencies. They cast themselves as young dreamers working 
to creatively inspire other young minds in India. When we initially contacted them 
we said, “We want to make a film about your rover, but using contemporary dance.” 
They replied “So cool!” 

Simultaneously we began a search for an Indian dancer who could translate a 
science story into a personalised performance. Emma Gladstone (Artistic Director 
Dance Umbrella) [11] recommended Hemabharathy (Hema) Palani. Hema was based 
in Bangalore, as were TeamIndus. From this point our themes developed. We discov- 
ered that TeamIndus’ robot, as often with ships or craft, was referred to as ‘she’ by 
the team who made her. TeamIndus co-founder Sheelika Ravishankar told us “When 
I received your proposal the first thing I asked was, is your dancer female?” Hema 
told us that the one-way nature of the robotic moon mission reminded her of a com- 
mon Indian female experience; when a young woman moves from her birth family’s 
home to the new home of her husband, it is sometimes said that “She will only ever 
return as ashes.” A parallel story began to emerge. We would track the journey of 
an Indian robot on the moon echoed by an Indian woman’s journey on Earth. Our 
filmic mise-en-scéne became Bangalore; once a small colonial outpost, now rapidly 
transforming into an Asian information technology-driven megalopolis. A city con- 
taining both ancient tradition and a world-beating aerospace industry. Our project 
became a container for exploring a dialogue between these two ideas. Figure 14.1 
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Fig. 14.1 R&D Studio 2018. Evocation of the cosmos created in milk droplets 


shows a frame of an animation rendered in development workshops, and eventually 
used in large-scale projection within live performances in Bangalore and Dorset. 


14.2 A Changing Real-World Context 


One of the challenges of documentary-making is access. The subject matter of the 
Indian Space programme could have presented us with several obstacles. ISRO [12], 
the governmental space agency responsible for nearly all of India’s activities in space, 
are not known for being an open access organisation and we had heard rumours that 
they were very unlikely to allow foreigners to visit them. We made several attempts 
to contact ISRO and received no response. 

Finding TeamIndus was a blessing as they were wonderfully open and generous 
and once in India facilitated our visits thoroughly and gracefully. However, we did 
not know if we would be able to find empathy between our very different professional 
disciplines. We spent a lot of time preparing our questions and equally allowed our 
interviews to unfold in a relaxed and unhurried atmosphere. When we first met 
TeamIndus in 2017 many teams from other countries had dropped out of the Lunar 
X-Prize and TeamIndus were ranked as favourites among the surviving four. It had 
been a white-knuckle ride with many delays and last minute deadline extensions. 
In January 2018, TeamIndus announced that they were scheduled to launch their 
spaceship in March-April that year. However, it was not to be. TeamIndus had to 
cancel their contract with ISRO (Indian Space Research Organisation), their launch 
provider [13]. It was a heart-breaking moment for the team. Google announced the 
closure of the Lunar X-Prize without a winner. However, it was clear that the idea 
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had been timely and had stimulated imagination in the private sector. A slew of new 
international missions to the moon were announced during this time, by national 
agencies such as NASA and ISRO and also many private companies around the 
world. 

In March 2019 the ‘new space race’ dramatically intensified. Ina speech at NASA’s 
Marshall Space Flight Center in Huntsville, Alabama Vice-President Mike Pence 
announced, in the most nationalistic terms, the U.S. government’s challenge to NASA 
to “return astronauts to the moon within the next 5 years.” Also saying, “The United 
States must remain first in space in this century as in the last. Not just to propel 
our economy and secure our nation, but above all because the rules and values of 
space, like every great frontier, will be written by those who have the courage to 
get there first” [14]. Also in March, Narendra Modi declared India to be the ‘Fourth 
Space Superpower’ after ISRO successfully destroyed one of its own satellites with a 
missile launched in space. Creating an estimated 6,500 pieces of space debris (in an 
exact echo of the opening sequence from the film ‘Gravity’) Modi announced a day 
of national celebration and called the action something that would “impress India’s 
rivals” [15]. In this year too the Trump administration announced the creation of a 
5th branch of the United States armed forces called the ‘space force’ [16]. In April 
2019, Israel’s Space IL, also a competitor in Google’s Lunar X-Prize, successfully 
launched the world’s first private moon mission on an unusual orbital trajectory. They 
proved just how challenging lunar expeditions could be as the craft unexpectedly 
crash-landed on the moon’s surface [17]. 

On 31 May 2019, TeamIndus tweeted that they had successfully partnered with 
American company Orbit Beyond and were once gain hoping to launch in 2020 as 
part of NASA’s CPLS programme [18]. On 15 July 2019, ISRO was set to launch its 
most complex lunar mission and hoping to make India only the 4th nation to soft-land 
on the moon. They cancelled due to technical difficulties only 56 min before launch 
[19]. The mission successfully launched 2 weeks later, but tragically communication 
with the lander was lost just before touch down when the craft was reported to be 
just 2.1 km from the lunar surface [20]. As of writing the craft is presumed to have 
crash-landed, demonstrating perhaps, together with the Israeli attempt, the scale of 
the technological challenge involved in reaching earth’s closest neighbour. 

The dramatic twists and turns of the story have meant that our project also has had 
to be fleet of foot, able to respond quickly to an ever-changing back-story. Within each 
development, and beside the grandiose and gung-ho statements of some players, we 
feel that our story offers up an Indian female human perspective, the ‘small dream’ 
of hope, set against the backdrop of an escalating international space race. 
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14.3 Short Film 


Ek Choti si Asha (A Small Dream) Dance film, 4 min, Channel 4 Television 2017 


The overall project had to meet considerable challenges: creative, technical and 
organisational, with many aspects that we had not tried before. The collaborations, 
though exciting, presented us with potentially difficult paths to negotiate. We first met 
Hema in Verbania north Italy, to watch her perform her own choreographic pieces 
in a dance festival. We were hugely impressed by her work, delicate and edgy at the 
same time. 

The first iteration of the project was a short film commissioned for Channel 4 
television; Ek Choti si Asha (A Small Dream) [21]. Our collaboration, including 
costume design, was developed across continents and time zones via Skype and 
WhatsApp. It was necessarily quick as we had limited time to develop the work. 
We designed a container for Hema to contribute to, sending video storyboards to her 
whilst she sent rehearsal clips of ideas and movements. Decisions were made quickly 
and the planning was precise. We attempted to present the audience with a deliberate 
conceit—that the inclusion of one narrative thread would be appropriated to explore 
another. In this case, the voices of young Indian space engineers would be featured 
describing how their robot would cope with physical challenges and obstacles on 
her perilous journey. This audio would be set against imagery of an Indian female 
traveller negotiating the busy streets of Bangalore; people, animals, traffic and the 
male gaze; the combination of voice and choreography allowed us to explore parallels 
between the Earth and Moon stories without literal space or moonscape being shown. 

Further to this idea, we decided to incorporate TeamIndus’ use of the feminine 
pronoun and request that they always refer to the robot as ‘she’. Interestingly at 
this point the two young male engineers warned us “You won’t get an emotional 
response from us. We are engineers and technicians.” They then proceeded to make 
recordings that were infused with obvious emotion as they described the robot that 
they had been creating for the past six years. 

We continued our remote video dialogue with Hema, sending her animatics (edited 
voice-over with improvised visuals) with Hema responding with small sections of 
dance recorded in her studio. In this way the ideas and the interplay between words, 
dance and imagery evolved and a larger narrative emerged. 

The specific aspects of the mission meant that the robot rover, named ‘Ek Choti 
si Asha’ (E.C.A. for short), meaning ‘a small dream’ in Hindi, would only survive 
whilst her solar panels could be in direct sunlight. One lunar day is equivalent to 14 
earth days. ECA would travel from Earth to Moon with her batteries switched off. 
She is only powered on when her wheels touch lunar soil. From this moment the 
engineers would be able to control her movements on the Moon from the Earth for 
a fortnight [22]. 

For us it meant we could give Hema a structure for her choreography. The young 
character ECA would begin asleep in a black space then wake and start her journey 
outdoors (Fig. 14.2). A dawn to dusk framework in the 4-min film would inform the 
remaining narrative that also included the entire lifespan of the character. At the end 
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Fig. 14.2 R&D Studio 2017. Ek Choti si Asha. Opening shot in Bangalore residential street, where 
Hema starts her walk. https://vimeo.com/240144999 


of the day, she faces the future with the knowledge that her end is imminent. Hema 
introduced tropes from marriage rituals [23] and developed her female character’s 
narrative, conceived of her as a contemporary Indian woman, yet bound by tradition 
and family. Simply told, she would leave home, full of hope as a young woman 
embarking on married life, yet she faces anxiety when she realises that her freedoms 
must change forever and she can’t go back. 

The storyboard and animatic followed this format, building mini-scenes or chap- 
ters based on different technical challenges that the engineers described in detail. 
Individuated sections were based on how to move the robot in low gravity, how to 
negotiate obstacles such as rocks and gradients, and how to remain safe from very 
fine electro-static moon dust. These factors meant that the engineers estimated they 
would be able to move ECA no faster than 5 cm/min. In response Hema’s first task 
was technical too—how to choreograph a walk so slow it resembled time-lapse. It 
was pleasing when we filmed tests and rehearsals outdoors that Hema’s choreogra- 
phy appeared to resemble slow motion, nicely belied by the figures around moving 
at normal speed. 

This first iteration enabled us to start building a film language that blended doc- 
umentary interviews with choreography and interwove a journalistic story with a 
fictional character. Ground rules were established and a simple narrative framework 
was conceived. For all the constraints of time and distance the simple structure 
worked very well and all three artists were satisfied with the result. 

For Hema the project also became expansive. During 2018 we began working in 
the context of a larger framework beyond the short film and a departure into live 
performance. 
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14.4 Outdoor Performance 


A Little Big Dream. National Gallery of Modern Art, Bengaluru January 2018 


In January 2018 the British Council in India was in the process of concluding a year 
of cultural events marking 70 years of Indian independence. Luke Jerram’s Museum 
of the Moon sculpture [24] was brought to Bangalore; a 7-m diameter spherical 
sculpture printed with high-resolution NASA photographs of the Moon’s surface. 

Hemabharathy Palani and R&D Studio were invited to make a performance 
beneath this artwork on the night of the “super moon’ on the 31 January 2018 outside 
the National Gallery of Modern Art [25]. 

The performance outdoors under the enormous moon sculpture immediately gave 
a sense of scale and suggested a duality of strength and fragility: a hardy robot built 
to withstand other worldly forces and yet tiny against the vastness of space. Hema’s 
choreography included a team of about 20 dancers who made a chorus surrounding 
her. She, however, was still alone, marooned on an island in the centre of an artificial 
pond, visually separated from the other performers. The audience were given an 
image of a lone explorer supported by a big team who were nevertheless separated 
from her and less and less able to control her as she journeyed further away. 

Luke Jerram’s Moon and the central performance were both reflected perfectly in 
the still water around her, a material existing in a non-solid state; an ethereal inverted 
echo transposed within another atmosphere and space. Pre-recorded video close ups 
of Hema’s movements projected into the decorative masonry and structure of the 
museum echoed her live performance, yet were strangely remote and disconnected 
too. The engineering team growing ever distant echoed a fading connection with 
birth parents. These key images expanded our metaphorical lexicon and represented 
several ideas at once (Fig. 14.3). 


14.5 Objects and Projections 


Residencies: Leverhulme & 101 Outdoor Creation Space Spring/Summer 2018 


Hema was invited by Ballet Rambert to develop new work during a residency spon- 
sored by Leverhulme [26]. Together with theatre producers Fuel [27], we started 
to develop ideas for a larger outdoor show, both in Rambert Studios London, and 
by securing a further ‘Seedbed’ residency at 101 Outdoor Arts Creation Space near 
Newbury [28], a large rehearsal space intended for artists to develop outdoor work 
at scale. 

In order to expand the film story into a live performance; narratively, choreograph- 
ically and visually, this more open-ended collaboration presented more challenges 
in terms of balancing our creative voices with our different cultural backgrounds, 
artistic training and preferred methodologies. 

We wanted to further develop the work as site-specific location-based, working 
outside to enable extreme contrasts in scale, the tiny robot in amongst the vast cosmos. 
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Fig. 14.3 R&D Studio 2018. A Little Big Dream. https://vimeo.com/2563 14451 


During these residencies, we conceived of making a wordless visual theatre show 
that would bring an Indian dimension to 2019, the year of the fiftieth anniversary of 
Apollo 11. 

During the Leverhulme residency we made a series of physical experiments, 
inventing mechanisms to create a sequence of images to carry a basic narrative. If 
we were aiming to tell a parallel story between a robot travelling from Earth to 
Moon and a woman moving between houses on Earth then perhaps simple contrasts 
could serve as starting points. We experienced a break-through when we began to 
play with objects and projectors in a darkened space. Choreography could be very 
small, just hands and small objects, while projections could be huge and immersive 
suggesting space or other worlds. We imagined how to combine content made with 
high technology with other images made with domestic objects, playing with large 
and tiny projections, with the body and with objects on a table-top (Fig. 14.4). 

During the 101 Residency, to increase our emotional range we invented a second 
character, a ‘mission controller’ who would appear on stage both as a live musician 
and a kind of director, tracking Hema’s character (who we called ‘Chandini’—trans- 
lating as ‘a bride for the moon’), and arranging small domestic objects on a table-top. 
For example, a cup of tea would become a surface for small focused projections that 
would be re-filmed and re-projected on a large scale thus suggesting immense objects 
or vast landscapes. Technical and visual experiments such as this were combined with 
narratives, expanded to take place on three stages: ‘Earth’, ‘Mission Control’ and 
‘Moon’. The action would take place live in an outdoor setting in front of an audi- 
ence but also filmed and projected on three large screens. The Earth and Moon stages 
would also have cameras directly above looking down, and the Moon stage would 
be covered with water. 
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Fig. 14.4 R&D Studio 2018. Table Top Moon. Leverhulme residency film. https://vimeo.com/ 
266294389 


We considered the potential use of a drone above the performer and audience, used 
to stream live images of Chandini’s journey to a large-scale outdoor audience. We 
took drone footage above Greenham Common around 101 Creation Space which, 
when treated, appeared like an alien landscape. (This footage was later used for 
subsequent stage iterations.) 

Via digital interactive tools a nuanced relationship between Chandini and her 
maker/controller/family became a further theme explored. Chandini arrives with a 
live camera attached to her head, whose feed is received and augmented by the 
mission controller and streamed onto the big screen for the audience. 

Interactive sensors attached to the dancers’ wrists created a cascade of ceremonial 
floating flowers symbolising marriage vows (Fig. 14.5). 

We mapped out a partially visualised narrative for a full-length outdoor piece in 
the form of a diagramatic triptych film incorporating spoken and labelled description 
with footage gathered from both residencies. Chandini Animatic (Fig. 14.6). 


14.6 Choreography 


Chandini—Work in Progress Ranga Shankara Auditorium Bangalore, Attakkalari 
Interim Festival, 2 February 2019 


Having focussed on technical possibilities the previous year, it was now time to 
develop the choreography. Hema worked on the identity and story of the character 
‘Chandini’ with several days of devising alone before our arrival in Bangalore. The 
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BIG SCREEN EARTH STAGE 


Fig. 14.6 R&D Studio 2018. Chandini Animatic. Triptych diagramatic film. https://vimeo.com/ 
297342932 


approaching public showing focused our energies and brought together the choreo- 
graphic and visual aspects of the piece. Hema’s choreography comes from working 
closely with music and we collaborated with PK a young Bangalore-based musi- 
cian who formed a series of digital soundscapes. In rehearsal sessions we mapped a 
story of Chandini’s journey on the moon; from the dream of her landing, adapting to 
her unforgiving environment physically and psychologically, her fears and struggles 
with a perceived feminine role, and her ultimate realisation that she is facing her new 
world alone. 

In the theatre film, animation sequences (e.g. Fig. 14.1) were projected onto dark 
surfaces and in ultra-wide-angle, bathing the whole auditorium and extending beyond 
the stage, exploring the impact of moving images as a vast spatial environment into 
which the small Chandini struggles to find her place (Fig. 14.7). 
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Fig. 14.7 R&D Studio 2019. Chandini—Work in progress. 20 min dance performance. https:// 
vimeo.com/320478323 


14.7 Festival 


A Small Dream at Dorset Moon Festival June/July 2019 


R&D Studio was invited to propose ‘Under The Moon’ ideas for ‘Dorset Moon’ 
Festival in summer 2019, a special one-off festival to coincide with the 50th anniver- 
sary of the Apollo 11 moon landing. Dorset Moon was curated by three partners in 
Dorset; Activate, producers of the Inside Out Festival, Dorset, Bournemouth Arts by 
the Sea, and b-side [29]. 

Using three spectacular settings we created A Small Dream [30] to showcase the 
story of ‘Chandini’ in three chapters: Launch, Journey and Arrival, comprising two 
outdoor performances and a video installation across 3 consecutive weekends in July 
2019. 

Each show was different and each was intended to work alone. By viewing all 
three the audience were able to track the story of a tiny female robot on a perilous 
400,000-km journey. Families, walkers, beach lovers and culture vultures encoun- 
tered a perambulatory dance through seaside gardens, films projected to look like 
stained glass windows and a final performance in a fort built out into the sea. The 
festival was the project’s most ambitious outing so far and once again we diversified 
our approach, developing and extending our metaphors. 

Hearing that ISRO’s Chandrayaan robot mission was planned for the same summer 
we also updated Chandini to embody her perspective within this current bullish and 
intensifying commercial race. 
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14.8 Live Art 


Launch Bournemouth beach and St. Peter’s Church 28-30 June 2019 


Chandini’s Dorset journey began on Bournemouth beach. It was the first sunny 
weekend of the season and the hottest day of the year thus far. People thronged to the 
beach. In amongst them squatting close to the pier Chandini gazed out to sea, a Go- 
Pro camera strapped to her head, recording her view. She gathered sand in her hands 
and made a ritual offering, a puja, to the sea. Slowly standing up Chandini began to 
pick her way across the sand, her slow cautious progress echoing that which might be 
taken on an alien surface. The imagery of the packed beach, as seen from Chandini’s 
headcam, is extraordinary. The camera picks up a strange world in which people are 
gingerly adapting everyday movements to an unfamiliar environment. Both staring 
and pretending not to look, the pleasure-seeking beachgoers are strangely unclothed 
and vulnerable. Chandini slowly passes through crowds, up close and intimate, pro- 
voking curiosity with a slow and mysterious approach towards Luke Jerram’s moon 
installed in St. Peter’s Church nearby [31]. 

After performing another puja at the door and taking off her shoes and headcam, 
Chandini enters the church to make a ten-minute performance under the moon includ- 
ing drawing the orbital path of ECA’s journey to the moon in a blue sand rangoli on 
the floor of the nave. 

After the performance audience members were asked, why do you think she is 
going to the moon? Echoing the question, why are ‘we’ as a species heading back to 
the moon after 50 years? 

The main form of this piece was not drawn from dance or theatre but from Live 
Art. Durational, interactive and placed in the real world this medium situated our 
character again in a new context and with a new relationship to an audience who may 
not even know they are witnessing an artwork (Figs. 14.8 and 14.9). 


14.9 Video Installation 


Journey Sherborne Abbey [32] 5-7 July 2019 


We next encounter Chandini’s journey installed in the magnificent eighth century 
Sherborne Abbey [32] in video projections in the shape of stained glass windows. 

In the quiet devotion of the ancient space, film imagery (such as a turmeric hand 
clap in slow motion and the cosmos as suspended milk droplets) was combined 
with the beach point of view scenes from Chandini’s Bournemouth headcam, and 
embedded into the crenulations of a ‘blind window’. 

Headphones played a musical score peppered with audience responses captured 
during the week, with the aim of subtly drawing viewers into current conversations 
about space exploration. 

The technique used here was projection mapping. Two projectors carried synced 
video imagery mapped into eleven complicatedly shaped windows. The venue echoed 
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Fig. 14.9 R&D Studio 2019. A Small Dream: Launch. Hema’s headcam POV 
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Fig. 14.10 R&D Studio 2019. A Small Dream: Journey 


Hema’s ritualistic and ceremonial interpretation of the scientific narrative. We almost 
literally created a window onto another world (Figs. 14.10 and 14.11). 


14.10 Outdoor Dance Theatre 


Arrival Nothe Fort, Weymouth [33] 12-14 July 2019 


The final part of the Dorset Moon trilogy was the largest and most ambitious chapter. 
The setting of Nothe Forte, a nineteenth century military fort built on a promontory 
with sea on three sides gave a grandeur to our performance and we consciously 
wanted to provide a spectacle for the 300+ audience. 
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Fig. 14.11 R&D Studio 2019. A Small Dream: Journey (seen through arch) 


We placed the audience up on the ramparts looking down into the fort where Luke 
Jerram’s Museum of the Moon was suspended from a crane. A stage was built just 
off centre beside the moon and we immersed Hema’s live performance in a huge 
projection extending 27 m across the circular floor of the fort (Fig. ). 

Finally arriving on the moon Chandini appears, small and fragile, in the centre of a 
cosmos projected at scale over the sunken grounds of the amphitheatre-like fort. The 
audience looks down on her as she wakes in an alien world surrounded by danger. 
Chandini confronts and struggles with her new environment whilst we realise she is 
ultimately abandoned. 

Being the grand finale to our constructed story developed over the three week- 
ends this piece was more akin to theatre. Using rehearsed choreography, projection 
mapping onto a square stage and within a larger elliptical surface, lighting and a 
live camera effect, the techniques were interwoven into the narrative. In addition to 
combat the prevailing sound of a windy environment, the audience wore wireless 
headphones so they could hear the soundtrack including its subtleties and speech. 

This performance presented large challenges as we moved from the medium of 
film to large-scale outdoor site-specific theatre and a variety of technical difficulties 
had to be overcome. The greatest challenge, however, was narrative, how to convey 
our emotional story also based on the current real lunar Indian mission, in the context 
of a family-based festival marking the 50th anniversary of Apollo 11. We tackled 
this with soundtrack; the performance begins with a cacophony of sounds and voices 
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Fig. 14.12 R&D Studio 2019. A Small Dream: Arrival. View of Nothe Forte performance from 
ramparts 


including current bold statements of ambition from national agencies, contrasting 
this later with the voice of TeamIndus’ Sheelika Ravishankar describing the diurnal 
cycle on the moon alluding to the emotional journey of the character Chandini. 


14.11 Documentary 


The Vyomanaut 30 min film October 2019 


Figure 14.13 shows a frame from the film Vyomanaut. 

Whilst making the choreography in India we began to record and document our 
process behind the scenes (Fig. 14.14). We also interviewed Hema about her feel- 
ings towards the work and also about her life in general. We felt as if we had an 
extraordinary resource in her. Fiercely independent and fearless, she did not have 
an academic background and doesn’t consider herself an intellectual. Not from an 
artistic family her first love was sports. Yet we found her to be a gifted creative and 
naturally articulate. She talked about her parents, how her father was supportive of 
everything she did but how her mother approved of her traditional performances but 
had refused to see any of her contemporary work. Almost constantly on tour, travel- 
ling the world and performing extraordinary edgy works that she had created herself, 
Hema is a cultural warrior steeped in Indian tradition and mythology. It was apparent 
that Hema herself embodied the same layers and complexities that the Indian space 
programme itself contained: simultaneously ground-breaking at the leading edge of 
innovation and entirely shaped by ancient culture. 
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Fig. 14.13 R&D Studio 2019. The Vyomanaut. Hema creating soundtrack. https://vimeo.com/ 
327902293 


Fig. 14.14 R&D Studio. The Vyomanaut. Super slow-motion shot of Hema’s feet landing in milk 
(the moon). https://vimeo.com/327502293 


The original conceit, the parallel story of a robot on the moon and an Indian woman 
journeying on earth, remained intact even though the film subtly shifts attention 
to our choreographer herself. In fact the metaphors deepened. Our documentation 
film became a portrait of Hema interwoven with a further exploration of the space 
mission. We visited TeamIndus again, with Hema, and gaining more insights into 
their evolving story, explored with them the parallels between their endeavours and 
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Hema’s creative world. Also, being appreciative of Hema’s love for speed, we made 
another visual encapsulation for the film: in contrast to her slow-motion walk, we 
filmed her dynamically running through Bangalore’s streets. 

The film allowed us to develop themes of speed, freedom for women in India, fam- 
ily, nature and science, wrapped up in a human portrait. It also exists as a document of 
our entire collaborative process from beginning to end, a video accompaniment to this 
chapter and a creative documentary film in its own right, incorporating different tex- 
tures of fly-on-the-wall documentary to highly rendered visual images (Fig. 14.14). 
It is currently being completed and will be pitched to dance film/science festivals in 
2020/21. 


14.12 Conclusions 


When we began making the Chandini project it was already a hybrid creature, mixing 
the worlds of contemporary dance, documentary film and animation. But we hadn’t 
planned for it to diversify into so many different forms. All three of our individ- 
ual practices are predicated on a playful intermingling of ideas and techniques and 
what we hope is healthy disregard for the traditional silos that some art forms find 
themselves constrained within. It is therefore unsurprising that our collaboration has 
become a multi-headed creature. 

We would argue that this process has not been schismatic but rather the opposite. 
The original playful conceit of the parallel story remains a constant, as does the 
overarching narrative of a one-way journey. 

Observing the engineers at TeamIndus search for innovative solutions to technical 
problems made us consider how adopting an expansive and exploratory approach 
could push our work forward in unexpected ways [34] and furthermore to ask—what 
is the purpose of creative research? 

By responding to layers within a story in the real world our iterations were driven 
by form following content, through deliberate meanderings across genre, form and 
technique, enabling us to examine the facets of our diamond in different lights, a 
challenging and illuminating process. 
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Chapter 15 A) 
Digital Moving Image Installations geai 
and Renewable Energy: 1994-2018 


Chris Meigh-Andrews 


Abstract In the period between January and April 1994, while I was artist in Res- 
idence in Digital Imaging in the School of Visual Arts, Music and Publishing at 
Oxford Brookes University, I developed Perpetual Motion, a gallery-based installa- 
tion presenting a computer animation powered via a wind turbine. This early work 
initiated a series of installations presented within a “white cube” gallery setting and 
outside in the landscape in which renewable energy systems were integral to the 
themes and functioning of the work and to the ethos and concerns of my approach to 
working with moving image and sound technologies. This chapter traces the devel- 
opment of a significant body of work produced spanning a twenty-five-year period 
from this initial project to some of my most recent installations, discussing my ideas 
and intentions; describing the functioning and operation of the work; and identify- 
ing my influences, context and approach as well as the challenges and issues that I 
encountered. 


Keywords Renewable energy - Solar energy - Wind turbines - Video art - Video 
installations - Video sculpture - Site-specific installations + Computer-generated 
images 


15.1 Introduction 


Throughout the period that I have been engaged in making moving image and sound 
installations, which includes all the works under discussion, I have aspired to produce 
works which are temporary and ephemeral. In every case, the installations involved 
the use of readily available objects and materials that were assembled to perform 
and function for a brief and finite period of time, usually simply for the period of 
the exhibition. The installations were subsequently disassembled and dismantled; 
the equipment and materials were reused or recycled whenever possible. This ethos 
is central to the work and to my intentions and continues to underpin my approach 
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to working with renewable energy and to my engagement and use of technological 
systems within my work in general. 


15.2 Perpetual Motion 


Perpetual Motion (Fig. 15.1) was the first installation I made using a computer to 
produce and present a moving image. Previously images within my work had been 
originated using analogue video techniques, although I had increasingly made use 
of digital effects and non-linear editing systems during the post-production since the 
mid-1980s. Perpetual Motion was conceived for a “white cube” gallery space and 
intended to be viewed within what I considered at the time to be a sculptural context. 
As with my previous sculptural video installations Eau d’Artifice (1990-93), Stream 
Line (1991-94) and Cross-Currents (1992), I hoped the work would be encountered 
and interrogated conceptually, by which I mean that I intended visitors to become 
engaged with an exploration of the elements and components of which the work was 
constructed and to trace the “logic” of its presentation and functioning to reach an 
understanding of the ideas and concerns of the work. Over time and with hindsight, 
I have come to recognize that this idea was both problematic and presumptive, but 


Fig. 15.1 Perpetual Motion (1994): Installation for computers, wind machine, wind turbine, video 
projector and video monitor. Funded by Southern Arts, Oxford Photography and the Ontario Min- 
istry of Culture & Arts, Canada. (Visiting Artist Award). Copyright © Chris Meigh-Andrews, 
2019 
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during this early period, I believed my work was sufficiently grounded in funda- 
mental ideas and concerns as to be universally understood and decoded from “first 
principles”. 

A basic description of Perpetual Motion is as follows: a digital animation of 
a flying kite was produced using “Adobe Photoshop” and “Macromind Director” 
(animation software) and displayed as a continuously repeating loop on a ceiling- 
mounted DC powered CRT video monitor which was connected to a 12 V battery on 
continuous charge via a small wind turbine. The turbine was subjected to a continuous 
blast of air produced by a large industrial fan at the far side of the gallery, which in 
turn was plugged into a conspicuously located mains socket. A second computer- 
processed video sequence of swaying grass was projected onto the gallery floor via 
a ceiling-mounted video projector. 

This work was subsequently exhibited in three different venues: Oxford Brookes 
University and the Saw Gallery of Contemporary Art, Ottawa in 1994 and Castlefield 
Gallery, Manchester in 1996 with the same basic configurations and components, 
although each time requiring minor modifications due to the differing architectural 
circumstances and logistics of the venue. In all three iterations, it was of central 
importance that the wind machine produced a sufficient flow of air to turn the turbine 
and provide enough current to maintain the charge to the 12 V battery powering the 
CRT monitor displaying the kite sequence. This flow of air was also a significant 
physical presence in the gallery, both in terms of the sound and the force of the airflow, 
contributing to what I considered at the time to be the “sculptural” experience. 

In her perceptive discussion of the installation for the exhibition catalogue, the 
curator Lowena Faull identified many of the key ideas and tensions behind the work: 


Perpetual Motion uses space in an intelligent way- using the dimensions of the gallery to 
create a sculptural installation which poses a series of relationships between the viewer, 
the physical presence of the objects and the technologies at work in the piece. There is a 
flow of the viewer’s imagination as s/he makes associative leaps between the wind machine 
driving the turbine and the image of the kite on the monitor. Meigh-Andrews has left creative 
gaps in his work, so that the audience is left to create a simple technical narrative- how it 
all works- and to create a narrative of meaning within the work itself.....the effect is both 
meditative and engaging....This is a circuit of energy, imagination and visual representation 
through which the artist comments on the representation of landscape, the desire of both art 
and science to represent and imitate the natural world, to force from its disorder, structure, 
to take its structures and create a more perfect replica. If there is an implied synergy / 
dependency / inspiration between nature and the machine- there is also an implied critique 
of that relationship [1]. 


15.3 Fire, Ice & Steam 


In 1995, I was given the opportunity to develop a new work for the Middlesbrough 
Gallery while engaged in an artist’s residency in Cleveland, North Yorkshire. The 
installation of Fire, Ice & Steam (Fig. 15.2) occupied three linked spaces in the 
Middlesborough Art Gallery, and although following on from Perpetual Motion, I 
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Fig. 15.2 Fire, Ice & Steam (1995): Installation for solar panels, halogen lamps and LCD screen. 
Funded by Northern Arts and the UK Foundation for Sport and Art. Copyright © Chris Meigh- 
Andrews, 2019 


was keen to continue developing new work featuring renewable energy components, 
only one of the rooms in the exhibition featured solar panels. As the focus of this 
chapter is on the development of my work with renewable energy systems, I will 
restrict my description to this aspect of the installation. The display in this room 
consisted of four framed and wall-mounted photovoltaic panels, connected to a 12 V 
“deep-cycle” battery powering a miniature LCD screen, displaying a repeating time- 
lapse forward/reverse video loop of melting and refreezing ice cubes. The melting 
and reforming ice was arranged to represent the written phrase “that time”. Each 
of the four walls of the room containing a framed solar panel was lit by powerful 
spotlights timed to switch on and off alternately, so that the light would sequentially 
rotate in a clockwise manner around the space across the day during the period while 
the gallery was open. The inspiration for this three-part installation was related to 
the industrial heritage of the region, and I was keen that the work was understood to 
be a representation of the complex and dynamic relationship between energy, light 
and time. 

This work provided me with my first opportunity to explore the potential of solar 
panels as a sculptural element within a gallery installation and provided the initial 
inspiration for further research. However, although the challenge of developing a 
work using the energy produced by photovoltaic panels in Fire, Ice & Steam led to 
further ideas, other installations took precedence (Vortex: 1995 and Mind’s Eye:1996, 
see http://www.meigh-andrews.com/installations), and it was not until 1998 that I 
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obtained the opportunity and the funding to develop Mothlight, a new installation 
involving renewable energy. 


15.4 Mothlight 


My starting point for Mothlight (Fig. 15.3) was an apocryphal story about the ori- 
gins of the term “bug in the system” to describe problems associated with com- 
puter programming. According to the version of the story I came across, an early 
prototype computer system at the computer lab at M.I.T. in Boston developed an 
operational fault. Because the room-size machine’s electronic valves needed to be 
kept cool during the long hours of calculation, all the windows in the lab housing 
the machine had to be kept open and during the night flying insects had entered the 
room, adversely affecting the operation of the computer. The engineer’s log for that 
particular day included a description of the fault as being due to “bugs” in the appa- 
ratus, and according to the story the log had even included the insect in question—a 
moth, carefully preserved between the pages! The flying insects in Mothlight were 
computer-generated, intentionally making what I considered to be an ironic refer- 
ence to this bit of scientific history, alluding to the uneasy relationship between the 
natural and the technological worlds. 


Fig. 15.3 Mothlight (1998): Installation for solar panels, halogen lamps and CRT Monitors), 
Funded by North West Arts, with support from the University of Central Lancashire. Copyright © 
C. Meigh-Andrews, 2019 
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Fig. 15.4 Mothlight, Rich 
Women of Zurich, London 
1998. Copyright © Chris 
Meigh-Andrews, 2019 


The decision to produce a computer-generated artificial moth was inspired by 
this story, but it was also in keeping with my approach to representation in previous 
works. (For example Eau d’artifice (1990), an electronic fountain created from sep- 
arately recorded elements playing out over a simulated “day”; Streamline (1991), 
an artificial stream made of nine separate but related sequences pieced together to 
form a “narrative”, or the previously mentioned Perpetual Motion (1994) in which 
kite, clouds, sky and grass were electronically combined to suggest a landscape.) 
In these installations, themes of conflict and interrelationships between the natural 
and artificial—‘“the made and the born”; between technology and nature—were at 
the heart of much of my video work during this period. This review, published in an 
Italian issue of “Flash Art’, identifies this relationship between the natural and the 
artificial, suggesting an engagement with the transmutation of matter, and picks up 
on both my fascination with creating parallels between the flow of matter and the 
flow of thought that had been inspired by my interest in the ideas of the physicist 
and philosopher David Bohm (1917—1994) [2]. 


In his research, (Meigh-Andrews) grew increasingly interested in natural images transfig- 
ured through a series of manipulations which create an artificial, alchemical world. The 
installations of the 90s focus, lastly, on a fundamental topic — the physic flux and its paral- 
lelism with the mental flux, and the possibility that one activates the other. The installation 
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at Calci plays on this twofold aspect — some halogen lamps illuminate four solar panels 
which feed some monitors which generate some moths. The whole process is an infinite cycle 
that the spectator mentally builds through a linear series of logical passages. The thinking 
flux also establishes a connection among spatially discontinuous elements. Mothlight puts 
forward another characteristic also present in other works of the artist — the search for 
contradiction, specifically the “ironic” exploitation of alternative energy. The viewer that 
has patiently reconstructed the path of energy cannot miss the fact that the solar panels are 
fed by the halogen lamps [3]. 


My video installation work at this time involved both sculpture and the moving 
image in interrelationship. In Mothlight, I sought ways to highlight the interdepen- 
dence of the elements which were the core of the work—the repeating cycle of 
the fluttering moths and the suspended solar-powered video screens illuminated by 
the halogen lamps were all connected to form an interrelated cycle of meaning. The 
light was an important theme in this piece—illuminating, powering and conceptually 
connecting the images and objects within the work. 

As stated above, my interest in the relationship between technology and nature 
was a major concern. In Mothlight the use of “renewable resources” was intended 
to be subversive. The solar panels were not used to generate electricity but to act as 
passive conductors which were transducing light from the domestic mains power- 
point. In my thinking at the time, I felt that by inverting the “conventional” appli- 
cation of renewable energy with electricity, I was serving the poetic rather than the 
technological. 

The structure of Mothlight was an attempt to make a work that suggested both 
balance and movement, and I attempted to make the mobility of the linked elements 
clearly visible insofar as was practically possible. The various elements which con- 
stituted the work were held in a physical balance—the solar panels, the illuminating 
lamps, and the video screens were arranged in counterbalance, in an attempt to make 
the physical balance echo the conceptual balance of the interrelated elements. 

The whole work was also of course a play on the idea of a “mobile”, and I 
wanted to make reference to the image of the moth as an illusion of mobility. The 
animated insect was tied to a predictable and endlessly repeating flight path, tethered 
as much as the various functioning physical elements of the work were constrained 
by the trailing cables and its relationship to the forces of gravity, and this illusion 
of movement was at the heart of the piece. The video monitors were placed in such 
a way that the viewer accepted (even if she/he knew better) the possibility that the 
moth was flying across the gallery space—flitting from one suspended screen to the 
other, and this was reinforced by a panning soundtrack and by the movement of the 
animation through the illusory space of the TV screen. I also observed that it might 
be possible to see the fluttering moth sequence as a reference to the flickering origins 
of the film and video image itself and had aspirations that the work could be read 
as a set of “nested” illusions—starting with the flickering origins of moving-image 
technology and outwards through the illusion of movement via the mechanics of 
perspective (both sound and picture) to the illusion of the mechanical mobility of the 
sculptural form of the entire installation. 
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Mothlight was exhibited on three separate occasions: the Museum of Natural His- 
tory in Pisa, Italy; the Glass Box Gallery at the University of Salford, Manchester; 
and the Rich Women of Zurich, Hatton Garden, London. Each installation was pre- 
sented in the same basic configuration, but with minor modifications to accommodate 
the architectural and logistics of the different venues. Each gallery required some 
adaptations and adjustments—some technical, others more practical. For example, 
the Natural History Museum, located just outside Pisa in Calchi, was exhibited in a 
space with an ancient ceiling and required a special fixing to be installed to support 
the combined weight of the installation. In the Glass Box Gallery, the work could 
only be viewed from outside the gallery through large windows on four sides of the 
space. As the work was completely contained within this space, the buildup of heat 
from the halogen light sources caused a significant voltage drop from the solar panels 
necessitating the hiring of a portable air conditioning unit. At the Rich Women of 
Zurich, the length of mobile support arms had to be modified to fit the gallery space. 

In 2001 a variation of this installation was developed for the 291 gallery in Hack- 
ney, London, replacing the CRT monitors with data projectors displaying the moth 
flight sequences onto suspended sheets of translucent glass. The images of the moth 
were sent via infrared to the projectors, and the transmitters were powered via the 
solar panels, which I considered at the time added a further conceptual level to the 
work, making a feature of the video signal “flying” through space from its source 
to reach the screen. The scale of Mothlight IT was more substantial and the greater 
ceiling height in the 291 gallery provided an opportunity for a more monumental 
installation, while retaining its ephemerality. 


15.5 For William Henry Fox Talbot (The Pencil of Nature) 


In 2002, I had the opportunity to realize a project that I had been considering for 
several years. In response to an invitation to make a new work for “Digital Responses” 
a group exhibition at the Victoria and Albert Museum, London, I proposed to install a 
solar-powered live video camera at Lacock Abbey, in Wiltshire—the former home of 
the pioneering scientist and inventor William Henry Fox Talbot. Electricity produced 
from a solar panel was harnessed to power a digital video camera focused on the 
large latticed window which had first been photographed by Fox Talbot in August 
1835. The image from the camera was composed to exactly reproduce Fox Talbot’s 
pioneering “photogenic drawing”, the world’s earliest surviving photograph. This 
digital facsimile was relayed via an ISDN phone line to the V & A, the resultant 
“live” digital image of the window presenting a full-size image of the window in 
“real time”. This digital replica of the oriel window at Lacock was presented in 
a special display beside an original copy Fox Talbot’s book, The Pencil of Nature 
(Longman, Brown, Green & Longmans, London, 1844), the worlds’ first book to 
be illustrated with photographs (Fig. 15.5). Researcher and curator Vince Dziekan 
described his encounter with the work when it was installed at the Victoria and Albert 
Museum in his essay “Distributed Aesthetics and the Tele-image”: 
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Fig. 15.5 For William Henry Fox Talbot (The Pencil of Nature), (2002): Site-specific installation 
for solar panel, video camera, ISDN phone line, and data projector.) A site-specific installation for 
“Digital Responses”, Victoria & Albert Museum, London, Sept-Oct. 2002. Curated by Paul Cold- 
well. Technical Consultants: David Dorrington (Internet) and Richard Monkhouse (Solar energy). 
Funded by The University of Central Lancashire, The Arts & Humanities Research Board and the 
London Institute with sponsorship from Canon UK, Solar Century PLC, with support from British 
Telecom, Bow Arts Trust and The National Trust. Copyright © Chris Meigh-Andrews, 2019 


At first, this fleeting projection could just as easily be dismissed as a case of the morning light 
outside being cast through the windows lining the length of this narrow gallery. Upon closer 
inspection, however, the shadow play seemed uncannily to re-enact the exact characteristics 
of this famous photographic image. To refer to this digital image as a representation seems a 
somewhat inadequate description in that the image gently playing on the wall surface I was 
facing involved the direct transmission of the light passing at that very moment, not through 
the windows in the very room in which I was standing, but through the actual windows of 
Lacock Abbey in Wiltshire, near Bath in the west of England. Titled, For William Henry Fox 
Talbot (The Pencil of Nature), the work was an exact re-composition of Fox Talbot’s famous 
‘photogenic drawing’, here captured by a solar-powered digital camera and relayed ‘live’ 
via an ISDN phone line to the gallery in South Kensington, where it was presented at actual 
size in ‘real time’ [4]. 


With this installation, I wanted to imply the complex web of interrelationships 
between art, technology, light, time and physical space and reference the origins 
of photographic imaging and the nature and significance of light and vision and 
its relationship to the flow of communication systems and to the interconnecting 
of two geographically separate sites. The work sought to build on ideas developed 
in previous installations such as Perpetual Motion and Mothlight which presented 
“renewable energy” as a metaphor. In this new work, the daylight at the site of 
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Fig. 15.6 The author setting 
up the video camera, Lacock 
Abbey, Wilshire. Copyright 
© Chris Meigh-Andrews, 
2019 


the abbey passing through the historic and culturally significant window at Lacock 
Abbey set the entire work in motion, reflecting my intention to present an installation 
in which the past, present and future were linked electronically, geographically and 
conceptually. 


15.6 Interwoven Motion 


At the edge of a wooded area in Grizedale Forest, Cumbria, overlooking Coniston 
Water in the English Lakes, a large tree was temporarily equipped with four video 
surveillance cameras arranged in a circular formation around the trunk at the height of 
approximately eight metres. The images produced by the four cameras were relayed 
via a switcher to a weatherproof LCD video display mounted at the base of the tree. 
The speed and direction of the camera image flow was determined by the velocity 
and direction of the wind. The entire system was powered by a wind turbine extended 
beyond the height of the forest canopy and four solar panels which were mounted 
within the tree itself (Fig. 15.9). 

Since beginning this period of work to develop renewable energy installations, all 
of the works I had developed were necessarily intended to be for interior locations, 
but for some time I had wanted to make an outdoor video installation that responded 
to its environment [5]. Working with video artist Catherine Elwes and engineer Dr. 
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Fig. 15.7 Richard 
Monkhouse and the author 
on the roof of Lacock Abbey. 
Copyright © Chris 
Meigh-Andrews, 2019 


John Calderbank in the early 1990s, I had conducted a period of research into the 
feasibility of building a permanent outdoor video sculpture for the Chiltern Sculpture 
Trail at Cowleaze Wood in Oxfordshire, although this early project did not proceed 
beyond the report stage [6]. 

The notion of constructing an outdoor video installation in the landscape con- 
tained many of the contrasting and contradictory aspects that I enjoyed working with 
at the time and which continue to some extent in my most recent works. Interwoven 
Motion juxtaposed the natural and the artificial and made use of technology which 
was intended for interior use placed outdoors. I was interested in finding ways to 
contrast the strength and fragility of the technology with the durability and vulnera- 
bility of the tree it was temporarily connected to and the landscape it was part of. I 
was also interested in highlighting and contrasting different notions of temporality, 
permanence and impermanence. The specific video images produced by the instal- 
lation were in themselves of no direct consequence—they were simply part of a flow 
of very subtly changing ephemeral moments. For me, the relationship between the 
light and the wind was at the core of the work. The light and wind provided the 
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Fig. 15.8 Interwoven 
Motion, Grizedale Forest, 
Cumbria, (2004). Copyright 
© Chris Meigh-Andrews, 
2019 


source of the images both in terms of the generation of the electrical power which 
supported the video and electronic apparatus and in terms of the direct physical and 
visual experience which become part of the work. (Day/night, ambient light and the 
movement of clouds, and foliage, the changing weather conditions, etc.) 

It should also be noted that the work itself, like the image-sequences it pro- 
duced, was transient. The components which constituted the work were temporarily 
clamped to a living tree for a period of ten days. The various bits of inexpensive 
technology—wind turbine, solar panels, video cameras, image switcher, LCD video 
display, cabling, etc.—were temporary modifications, which, once removed, left no 
trace. During the period in which the prototype installation was functioning, it was 
left running night and day for as long as the technical systems remained operational. 
Designed to be self-powering as long as the weather conditions provided sustaining 
light and wind, the installation was equipped with two large-capacity rechargeable 
batteries capable of powering the installation for approximately 72 hours. Located on 
Forestry Commission land, it was relatively inaccessible except via an unpaved road. 
From the distance, the solar panels and the wind turbine would certainly have aroused 
the attention of curious by-passers. However, the casual visitor coming across the 
installation would find no explanation or context for the piece, what it was, why it 
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Fig. 15.9 Interwoven Motion, (2004): Outdoor site-specific installation for four cameras, wind 
turbine and solar panels. Commissioned for “ITEM”, Funded by the National Endowment for 
Science, Technology and the Arts (NESTA) and Arts Council, England for The Foundation for 
Art and Creative Technology (FACT) Liverpool, with support from Grizedale Arts, Wind and Sun 
Ltd. and the Forestry Commission. Consulting Engineer, Dr. John Calderbank. Copyright © Chris 
Meigh-Andrews, 2019 


was there, or what purpose it might have. Visitors were free to respond (or not) and 
to offer up their own explanation for its existence. 

The location of the prototype outdoor video piece at Lawson Park was significant, 
as the site was on land once owned by John Ruskin, the influential Victorian English 
writer and critic. Ruskin’s passionate enthusiasm for the landscape of this area is well- 
documented, not least in his published lectures and prolific diaries. His detailed and 
evocative descriptions of the ceaselessly changing views of the “Old Man” above 
Coniston Water and of the cloud formations and vivid skies provided me with a 
compelling sense of this dynamic landscape—and provided me with a title for the 
work: 


From the west the wind blows fiercely towards you out of the blue sky. Under the blue space 
is a flattened dome of earth-cloud clinging to, and altogether masquing the form of, the 
mountain, known as the Old Man of Coniston. The top of that dome of cloud is two thousand 
eight hundred feet above the sea, the mountain two thousand six hundred, the cloud lying 
two hundred feet deep on it. Behind it, westward and seaward, all’s clear; but when the 
wind out of that blue clearness comes over the ridge of the earth-cloud, at that moment 
and that line, its own moisture congeals into these white—I believe, ice-clouds; threads, 
and meshes, and tresses, and tapestries, flying, failing, melting, reappearing; spinning and 
unspinning themselves, coiling and uncoiling, winding and unwinding, faster than eye or 
thought can follow: and through all their dazzling maze of frosty filaments shines a painted 
window in palpitation; its pulses of colour interwoven in motion, intermittent in fire,— 
emerald and ruby and pale purple and violet melting into a blue that is not of the sky, but of 
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the sunbeam;—purer than the crystal, softer than the rainbow, and brighter than the snow 


[7]. 


Ruskin’s vivid perception and appreciation of the lakeland landscape provided me 
with a powerful connection to the cultural history of the site and was an important 
element within the context of the work, deeply connected to a sense of its location. At 
that time, I saw this project as the first step towards the goal of creating a landscape 
installation that was an integral part of the landscape in which it was sited, responding 
directly to and in relation to its location, which remains an aspiration to this day [8]. 


15.7 Resurrection 


The subsequent invitation to produce and exhibit an installation incorporating renew- 
able energy was once again within a gallery context and I took the opportunity to 
develop a companion piece to the recently completed Interwoven Motion. A descrip- 
tion of the elements and structure of the installation may help to visualize the work 
(Fig. 15.10). 


Fig. 15.10 Resurrection, 
(2005-06): Site-specific 
installation for solar panels, 
DVD player and data 
projector. Commissioned for 
“Digital Discourse”, curated 
by Vincent Briffa, St. James 
Cavalier Centre of Creativity, 
during the Commonwealth 
Heads of Government 
Meeting, (CHOGM), 
Valletta, Malta. Funding 
from the British Council and 
support from the University 
of Central Lancashire. 
Copyright © Chris 
Meigh-Andrews, 2019 
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A dead tree, complete with roots (approximately twenty feet high), was cut in 
half, the root end mounted in the centre of the floor at one end of a rectangular 
gallery space. The upturned tree and roots were lit by halogen lamps, casting strong 
shadows on the opposite gallery wall. Over forty individual miniature solar panels 
were arranged irregularly on the roots and connected in series, the wires grouped 
and bundled and flowing down the trunk and in a cluster along the floor towards the 
centre of the gallery. The clustered cables were connected via a junction box and 
through a regulator to a twelve volt battery positioned in the centre of the space. The 
battery was connected in turn to a DC powered DVD player. 

The top half of the tree was mounted in the centre of the ceiling pointing down- 
ward, its branches reaching down towards the floor with numerous small rectangular 
sheets of heavy white paper fixed to the branches, arranged to resemble leaves. A 
data projector mounted on the ceiling at the opposite end of the gallery was fed a 
continuously cycling pre-recorded video sequence of the original living tree, com- 
plete with leaves stirred by a fresh breeze. The sound of the wind in the leaves filled 
the space and the projected image created a strong silhouette of the upturned tree on 
the gallery wall. 

This new gallery installation, entitled Resurrection, drew directly on the experi- 
ence of building Interwoven Motion, bringing both the technological and the natural 
elements back into the “white cube” gallery space to create a companion piece. The 
living tree of the Grizedale project was conjured up in a revivified form within the 
interior of the gallery. 

The video images of the fluttering leaves in Resurrection presented a record of a 
previously living existence, recreated via technology. The electrical energy used to 
bring the resurrected tree back to life was transformed within the gallery space from 
electricity to light and back again and the shimmering leaves were experienced as 
both light reflectors and light receptors, the solar panels as both surrogate leaves and 
transforming technology. 

Resurrection was developed from the knowledge gained through the development 
and making of previous works and shared many of the same conceptual concerns (and 
components). In retrospect, it was a refinement of my approach to previous gallery- 
based installations made prior to Interwoven Motion because it offered gallery visitors 
a more pronounced sculptural experience. The technical and functional aspects of the 
work were more integral to the spatial arrangement of the sculptural components. 
In this sense, Resurrection was more akin to my aspirations with Mothlight and 
Perpetual Motion and my earlier non-renewable installations such as StreamLine in 
which I was conscious of the relationship to the architectural space of the gallery and 
intent on achieving a balance between the metaphorical “function” of the technical 
components and a perceptual (haptic) experience within the exhibition space. 


286 Chris Meigh-Andrews 


15.8 Sunbeam 


The environmental burden of electronic equipment, in terms of resources, manufacture, 
energy use and waste has become a matter of concern for artists working with this equipment. 
Some have developed works that variously recycle old technologies or decline to take their 
power from the grid. Thus, Chris Meigh-Andrews’ SunBeam uses a solar-tracking array 
(photovoltaic cells that follow the sun’s arc during the day) to power a night-time projec- 
tion of processed images from NASA’s Solar Dynamics Agency, the content and the process 
matching. The piece appeals to a cultural history of solar energy and the symbolic power 
of plants, like the daisy and the sunflower, that track diurnal rhythms in their behavior and 
growth [9]. 


I had been fascinated by the large tracking solar array installed adjacent to the 
School of Dentistry at the University of Central Lancashire in Preston ever since 
they had been installed in 2009. This solar array was able to track the sun as it 
moved across the sky during daylight hours, substantially increasing its efficiency 
[10]. During discussions with Dr. Robert Walsh, director of research at the Jeremiah 
Horrocks Institute for Astrophysics and Supercomputing who was seeking ways to 
publicize the institute’s work with NASA’s Solar Dynamics Observatory, I suggested 
that we harness the energy from the solar array to produce a series of evening pro- 
jections of the high-definition images of the sun produced by the observatory. My 
proposal was to draw energy collected during the day to project images of the sun 
back onto the surface of the array itself in a reversal of the process. Tests during 
the development stage of the project revealed that this would require covering the 
panels with a reflective material during the evening to provide an image of suffi- 
cient brightness to be visible. The project would also require a high-powered data 
projector and local police clearance to allow the images to be projected from the 
building across a busy road from the array. For the screen, I ordered a custom-made 
PVC sheet large enough to cover the surface of one of the two solar panels and 
this was installed using a large hydraulic lift. At dusk on the evening of each event, 
the covered solar array was rotated to face the university building directly across 
the road in which a Christie high-definition video projector was installed on the 
third floor. Edited time-lapse high-definition video sequences were projected onto 
the array on four consecutive nights, drawing considerable crowds and providing a 
spectacular demonstration of the power and majesty of the sun and of the university’s 
collaborative research initiatives (Fig. 15.11). 

My own aspirations for this work extended beyond this and were related to ideas 
and concerns that link it to my previous renewable energy installation projects. 
As Charlie Gere, professor of media theory and history at the Lancaster Institute 
for the Contemporary Arts at the University of Lancaster, has observed, SunBeam 
brought me closer to my conceptual goal of producing a technological artwork which 
attempts to integrate the source of its energy with the images it presents, to celebrate 
the harmonious relationship between light, energy and the fluid nature of matter in 
general: 


In SunBeam Meigh-Andrews now perhaps realises what the earlier works hinted at, an 
artwork which both represents the prodigious energy of the sun and performs its effects by 
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Fig. 15.11 SunBeam, (2011): Site-specific digital projection event. (Tracking Solar array, data 
projector.) Produced in collaboration with the Jeremiah Horrocks Institute for Astrophysics and 
Supercomputing, UCLan and the Solar Dynamics Observatory, (NASA). Scientific Advisor: Dr. 
Robert Walsh, with assistance from Dr. Stephane Regnier, David Henckel and Michael Dorricott. 
With support from Astley Hire. Video editing of solar sequences: Cinzia Cremona. Copyright © 
Chris Meigh-Andrews, 2019 


using that energy to make the representation possible...That the energy harvested during 
the day can then be used to make an artwork possible beautifully encapsulates (Georges) 
Bataille’s notion of art as a form of general economy exemplified in the sun itself. The system 
that harnesses the sun’s extraordinary power for straightforward and restricted uses, such 
as supplying energy to the university and to the national grid, is ‘detourned’ to produce 
a work of art, or in other words something useless according to the restricted economy of 
reciprocity and exchange. This is, perhaps, the very definition of art itself. [11] 


15.9 Aeolian Processes 


Aeolian Processes I and II (Fig. ) were unusual within my body of work in that 
renewable energy was harnessed to produce sound and neither installation included 
any visual imagery [12]. However, both works followed my general approach of 
seeking to produce a series of visual connections enabling the functioning of the 
work to be decoded by following the operation of the elements from which the work 
is constituted; hence the subtitle of the 2nd version of this work “Box Revealing 
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Fig. 15.12 SunBeam. Installing the temporary screen on the solar tracker, UCLan, Preston, May, 
2011. Copyright © Chris Meigh-Andrews, 2019 


the Sounds of its Own Making”, which makes a reference (and homage) to Robert 
Morris’ 1961 sculpture [13]. 

Both installations had a definite physical presence and the various technical com- 
ponents that produced the sounds were all visible and involved elements of movement 
and change. As with all the installations discussed in this chapter, I was interested in 
making a temporary sculptural object in which the visitor engages with the function- 
ing and operation of the work in order to make sense of it. With Aeolian Processes, 
I aspired to make something which was simple, direct and compatible with the land- 
scape ethos of its location. Both installations were comprised of similar elements 
and operations, although Aeolian Processes II is a refinement and development of 
the earlier work. 

Aeolian Processes I was commissioned for a group exhibition to be sited in a 
large urban park, and consisted of a large wind chime suspended within the interior 
of a glasshouse which was being continuously activated by the breeze from a small 
domestic fan. A microphone placed close to the wind chimes picked up the sounds 
which were then amplified, electronically manipulated and relayed to an outdoor 
speaker installed on the roof of the glasshouse. All the electrical components were 
powered via solar panels which were also installed on the roof. 

Aeolian Processes II (Box Revealing the Sound of its Own Making) used the same 
basic configuration of components as the previous version, but given that the work 
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Fig. 15.13 Aeolian Processes I (2013): Outdoor solar-powered sound installation, commissioned 
for “Art in Your Park”, Highfields Park, Nottingham, curated by Theresa Caruana, with funding 
from Arts Council, England. Aeolian Processes I (Box Revealing the Sound of its Own Making) 
(2014), “La Lune: Energy Producing Art’, Long Reef, New South Wales. Curated by Allan Giddy 
with funding from the Environmental Research Initiative for Art, College of Fine Arts, University 
of New South Wales, Australia. Copyright © Chris Meigh-Andrews, 2019 


was commissioned for a sculpture trail intended to be viewed after dark, the interior 
of the glasshouse was illuminated. As the work required a more substantial power 
supply, the solar panels were located on a separate structure which was positioned 
adjacent to the glasshouse. The electronics were also refined; an echo device was 
incorporated into the sound processing and an improved amplification of the wind 
chimes enabled the sound to be heard from a greater distance. 

In 2015, I began developing a series of small-scale sculptural installations under 
the generic title of “Impossible Objects”, motivated by an interest in making works 
which would not require a specific commission, purpose, venue or dedicated gallery 
space, and with no major funding. Although these pieces have much in common 
with earlier works discussed in this chapter in that they often incorporate or feature 
renewable energy components in order to make connections to the theme of flow, 
they are also more directly centred on notions of “process” as a central concern. 
They are “Impossible Objects” not because they cannot exist (as they clearly do), 
but because they make use of or refer to a process that contains a contradiction 
or presents an “impossible” idea. They are representations of a state or situation 
that cannot be achieved, except through the processes and agency of art. In this 
respect, I have been influenced in part by the Mono-ha works of the Korean artist Lee 
Ufan, in which there is an encounter between different materials—‘“a relationship 
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of tension” in which the work is the site of the encounter. In common with my 
approach to my large-scale installations, all of these “Impossible Objects” are hybrid 
installation/sculptures made using domestic technology; temporary assemblages are 
made using readily available materials and equipment. Most involve moving images 
and some have sound. Although there are at the point of writing five in the series, 
only two are relevant to the themes in this chapter. 


15.10 Impossible Objects 


Intended to fit onto a tabletop, Impossible Object Number 1 consists of a symmetrical 
arrangement of two solar panels top lit by two anglepoise lamps (Fig. 15.14). The 
panels provide a continuous charge to a 12 V battery which provides the electrical 
energy to drive a small electric motor geared to turn a crank operating a miniature 
music box mechanism which repeatedly plays the first few bars of the tune from 
(John) Lennon’s “Imagine”. The work employs a similar ironic reference to that 
of my earlier large-scale installations Mothlight and Mothlight IT, i.e. the notion 
that solar energy (or more accurately renewable energy as a whole) will solve the 
current environmental crises. Although the work is perhaps more directly critical 
when compared to the earlier works, it is also more whimsical. 

Similarly, Impossible Object Number 2 (Fig. 15.15) continues this more directly 
cynical perspective with respect to the misplaced optimism regarding the role of 
renewables to “save the planet”. Physically, the work is more of a sculpture than an 


Fig. 15.14 Impossible Object Number 1: Imagine (No Pollution) (2016-17). Anglepoise lamps, 
solar panels, battery, electric motor, music box. Funding: First Site Collectors Group Bursary, 
(2015). Copyright © Chris Meigh-Andrews, 2019 
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Fig. 15.15 Impossible Object Number 2: Blue Sky Thinking (2018-19). LCD screen, media 
player, wind turbine, battery and electric fan. Copyright © Chris Meigh-Andrews, 2019 


installation. A black 92 cm x 3 cm x 8 cm wooden beam leans against a gallery wall 
at a 45 degree angle, with one end sitting on the floor. At the bottom end nearest the 
floor, a compact domestic electric fan creates the air current to turn a toy wind turbine 
connected to a battery powering a media player connected to a miniature LCD video 
monitor, which is arranged so that it must be viewed through a magnifying glass. 
The monitor displays an endlessly cycling video sequence of a full-size wind turbine. 
The video presented on the screen was recorded using a rotating mount locked to 
the speed of the turbine rotors, so that the surrounding landscape appears to rotate 
while the turbine blades remain static. As with all the other installations discussed in 
this chapter, both works are conspicuously plugged into the gallery’s mains power 
supply. 

These two small-scale sculptures clearly draw on the ideas, techniques and 
approach of the earlier installations, and although they are less ambitious in scale 
and execution, they were intended to be more direct in their intention and content. 
They might be understood as maquettes, as it would be conceivable to create larger 
versions of both of these pieces, and at a larger size they might be more challenging 
to experience in a gallery or outdoor setting and perhaps even more effective. Certain 
elements would need to be reconfigured and the formal arrangements would require 
revising and modifying, and in an upscaled format both of these “sculptures” would 
almost certainly, in my view become “installations”. 
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Fig. 15.16 Impossible Object Number 2: Blue Sky Thinking (2018-19), detail. Copyright © Chris 
Meigh-Andrews, 2019 


15.11 Conclusions 


Although all of the works discussed in this chapter inevitably have a level of critical 
engagement with the impending current environmental crises, and is therefore rele- 
vant to any reading of the work, in very few cases was this aspect my sole motivation 
or even my central concern. The photovoltaic panels and wind turbines in my instal- 
lations and sculptures have been employed for their ability to transform or transduce 
energy from one form to another; from light or flowing air into electrical energy. In 
all of these installations, I have attempted to create or make reference to the flowing 
movement of matter and its parallel to the fluid movement of time. In relation to 
this, I wanted to find ways to create an awareness of the process of perception that 
takes place in the mind of the viewer while she/he actively engages with the work 
and participates in its potential to exist and/or function. 

It seems as if there is something to be said about the terms I have used to describe 
the various works. I have written about “installations” because I feel that most of the 
work I have developed that makes use of or features renewable energy systems are 
works which require the visitor or viewer to engage in a kind of perceptual action— 
reading or following a line of thinking and making links and connections between 


15 Digital Moving Image Installations ... 293 


elements to reach a conclusion or arrive at an understanding. This could be seen as 
a kind of narrative activity or at least a time-based process of reading and making 
connections. A sculpture, on the other hand, requires that it is seen and understood— 
or at least perceived “all at once” as an object, and then perhaps unravelled and 
deconstructed, after the initial encounter. 

At the point of writing this, I have tentative plans for further work to extend or 
continue my line of enquiry which may perhaps result in new or further successful 
developments. However, I believe the context of this work has changed considerably 
since I began making installations which included renewable energy components. 
The environmental issues have been brought into much sharper focus, as there is a far 
greater public awareness of the danger to the environment posed by the use of fossil 
fuels. For example, in my earliest work visitors were not always able to immediately 
recognize the function of the solar panels as transducers of electrical energy, but 
now these objects are so commonplace that they are not perceived as remarkable or 
intriguing and their potential as a symbolic or poetic device has been considerably 
eroded. This shift in consciousness makes the environmental dimension of the work 
too dominant, and weakens its impact and potential, and this requires that I either 
accept this and move on, or try to discover a new level of signification and relevance 
for them in my future work. 
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Abstract This work considers how the arts and technology in combination can stim- 
ulate connections in heritage gardens, and also nurture care for non-human nature. 
The chapter divides into two overlapping parts. The first part describes and critiques 
the design of Digital Nature Hybrid artifacts for interpreting gardens and exploring 
nature. The second part builds on the first by showing how the challenges presented 
by the Digital Nature Hybrids stimulated the design of Digital Naturalist workshops. 
It shows the value of combining arts, digital technologies and outdoor activities 
to support active engagements with non-human nature and to inspire the develop- 
ment of knowledge and skills needed to attend to natural environments. Research 
through design underpins the way of working and the project uses a critical approach 
toward technology, to guide the design decisions. One of the insights is the value that 
adopting this critical approach has in shaping both processes and designs. 
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16.1 Introduction 


Digital and other technologies play an increasing role in human life and have become 
the dominant paradigm in the industrialized world. Technologies shape our move- 
ment and interactions, for example, the way we walk through streets, head down, 
checking our phones. Each technology favors particular senses and pushes and pulls 
at them for attention. The values and assumptions that are built into technologies 
are osmosed into society and become accepted as norms by both individuals and 
wider society, hence shaping the cultural horizon. Once they have been normalized, 
technologies reproduce themselves because people take them for granted and are 
conditioned to notice and value the benefits they bring. 

Despite widespread acknowledgement of technology’s contributions, many writ- 
ers over the decades, from Simmel [1, 2] and Marcuse [3] to Feenberg [4-6] and 
Kahn [7, 8] have expressed concern at the effect of technology on humans and non- 
human nature. Some draw attention to the impact of technicity (the prevalence of 
and reliance upon technology) on the human psyche, others draw attention to the 
framing of the world that prevails in this paradigm. Common themes are alienation 
from the self, from others and from the natural world, and a tendency to perceive 
every thing and every person as a resource to be harnessed and operationalized, and 
an inability to see this happening from within the system. 

The implications are particularly evident in human relationships with the natural 
world. People spend more time in communion with their digital devices and less 
time in nature, leading to a loss of familiarity, awareness and care for the natural 
environment. As people’s care for a place dwindles, protection of that environment 
declines, and people care even less, leading to an exponential cycle of deterioration. 

Fortunately, the cycle can be broken, and the cultural horizon can be shifted, for 
the benefit of humans and non-humans. Feenberg [4—6] argues that technical codes 
can be reshaped through a democratic, participatory approach. His work builds on 
Marcuse’s [3] which looks to artists to critique and confront systems and structures. 

This chapter discusses digital designs which draw on these ideas, embracing crit- 
icism of technologies in order to design differently. The aim of these designs is to 
revive nature connections and re-energize the culture of amateur naturalists, to ensure 
that the detailed local knowledge needed to safeguard environments and biodiver- 
sity is not lost forever. The designs range from Digital Nature Hybrids intended to 
stimulate senses and raise awareness of gardens and plant heritage to workshops 
that meld arts, computing and nature encounters to raise levels of engagement and 
connection to the natural world. These workshops bring together experts from the 
different domains, and each iteration responds to location, participants and inputs 
from the particular partners. Beyond the individual workshop, the project’s aim is to 
strengthen a collaborative community of artists, designers, technologists, environ- 
mentalists and educators to excite people about exploring their local area and the 
wildlife around them. 
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16.2 Background 


The Industrial Revolution caused major changes to the economy and society that cas- 
caded down to individuals’ everyday life. People moved from the countryside into 
cities, becoming geographically separated from the natural world. The character of 
work changed. Agricultural work had flexed in time with natural rhythms of the day, 
across the seasons, but machine processes exerted a fixed, relentlessly unwavering 
pattern, increasingly uncoupled from rhythms in the natural world. Previously people 
made whole things, from start to finish, but the compartmentalized structures neces- 
sitated by machine processes fragmented production, which meant workers carried 
out discrete, repetitive tasks. Workers no longer saw the workflow from the start to 
finish and this had consequences. They lost the autonomy to manage their time, they 
became deskilled, and as a result, were less invested in the production process. 

Simmel contended that workers experienced a sense of disconnection in work that 
affected their identity and extended to their sense of self. The intensity of industrial 
life distracted and bombarded humans [1] to the point where they had no mental 
space for reflection and they withdrew emotionally from work and other aspects of 
life, including the natural world. The narrative of disconnection was later revisited 
by Marcuse [3] in “One Dimensional Man”, which described how technological 
structures overwhelmed life, to the point where anything that did not fit the dominant 
technological worldview was squeezed out. In this setting, every thing and every 
person came to be viewed as a resource to be harnessed and operationalized. This 
created a situation where people were unwittingly trapped and consumed in their 
never-ending pursuit of a lifestyle, associated with technological progress, which 
was promoted through advertising and marketing. 

Some, like Heidegger [9, 10], have asserted that technology has a particular charac- 
ter that determines these behaviors and it is so pervasive that retreat from technology 
is the only way to halt disconnection from the natural world, alienation from one’s 
self and damage to the psyche. Heidegger argues that technology induces a mindset 
in which everything is seen as “standing reserve”, a resource to be consumed, and 
the essence of things is lost or diminished by this mentality. This perspective implies 
technology is an external force that acts independently of society to determine partic- 
ular behavior and values. It has since been shown that technology isn’t acting alone 
in determining behavior but is heavily influenced by society [4—6]. Crucially this led 
to the possibility that digital technologies could be imbued with a different set of 
values, prompting alternative behaviors. 

This chapter discusses research that explores the use of a critical approach to 
the design of digital technology for interpretation of gardens and the natural world 
in support of connection to place and non-human aspects of nature. In Marcuse’s 
critique of technological society [3], he framed artists as outsiders, with the distance 
and perspective necessary to uncover and challenge the pervading conditions that 
limit human life. Some fields of art and design have become part of the (hegemonic) 
structures that perpetuate the cycles of desire that Marcuse criticized, exemplified 
by “affirmative design”, which serves the needs of industry without challenging its 
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structures [11, 12]. Critical design, by contrast, asks questions about society, its 
values and structures and critiques culture through the medium of design. This role 
has traditionally been the domain of art, so it has been argued that this pushes design 
toward artistic practice [12]. Dunne and Raby have been drivers of critical design, 
through projects such as Foragers, which asks questions about humans’ futures in 
the time of food shortage and overpopulation [13]. In recent years speculative design 
and more specifically design fiction, which challenge the status quo or ask “what if” 
questions about potential futures that have gained traction. 

This chapter describes work that is derived from the same theoretical roots as 
critical design but diverges in approach. Where critical design critiques society and 
speculates on futures through designed artifacts, the work discussed in this chapter 
attempts to design differently, within the parameters of the current world, by using a 
design lens that keeps the perceived problematic aspects of technology to the fore, 
while making conscious efforts to address these problems. This approach focuses on 
the implications of design process as well as design outputs. 

The remainder of the chapter discusses practice-based research conducted by the 
author that explores the use of a critical approach to the design of technology in 
the context of environmental communication. The research which began as the lead 
author’s thesis research [14] in 2013, and is ongoing as post-doctoral research, can 
be divided into three phases: a short initial phase of exploration and prototyping; 
a five-year collaboration with a team of gardeners at a National Trust property in 
Nottinghamshire; and development of thesis findings as a research strand within a 
program researching digital technologies and environmental change. The final phase 
has a primary focus on Morecambe Bay, Lancashire but its reach has expanded 
across North West England. The research marries the lead author’s background as 
a Secondary School Geography teacher and a Higher Education researcher, lecturer 
and multimedia designer. 


16.3 The Design Lens 


The design lens draws on recurrent discourse found in critiques of technology, as well 
as themes that are specific to particular theorists. Some of the original arguments were 
made prior to the ubiquitous presence of digital technologies, though they retain their 
relevance. Describing all the design principles and the theories that underpin them is 
beyond the scope of this chapter, but the following example shows the development 
of the design lens. 


16.3.1 Technologies, Things, People and Context 


One of the starting points is that technologies differ substantially in character from 
non-technological “things.” Devices, (Bormann’s term for technologies) [15-17], 
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are designed to be ubiquitous, so they can easily be used anywhere. The assets 
they provide are foregrounded, while the controlling mechanisms are hidden away. 
Without much thought or effort, they become absorbed into the background of our 
lives. The freedom devices bring comes with a hidden cost. Devices are (intentionally) 
detached from their context, in order that they can be used anywhere, by anyone but 
this means they are not rooted in a culture and its values [15-18]. Technological 
devices are a means of accessing/unlocking resources, be they reserves of fuel or 
information and as a consequence, they frame everything as a commodity to be 
consumed. Heidegger calls this “standing reserve” [9] and he believed that this lens 
prevents us from seeing things in their own right. 

By contrast, it is argued that (non-technological, handmade) “things” provoke a 
different response, because a thing is of its world and is connected to that world 
by a myriad of social, historical and material threads. Things are indebted to the 
people, place, environment, history, culture, knowledge and traditions that brought 
them into being and continue to sustain their life [19, 20]. When humans encounter 
a thing, they are momentarily “interrupted” by it. They pause to notice and reflect 
on it and the world that brought it into being. They see the thing in and of itself [20]. 
Where technology’s lens reveals everything to be a raw material to be harnessed by 
humankind, in an ongoing, never-ending process, things illuminate the world that 
surrounds them. Things are seen through their use, but technologies ultimately dis- 
appear in their use [19]. Spinosa and Dreyfus [21] suggest that if technical devices 
could behave more like other “things”, humans could be drawn into a positive rela- 
tionship without themselves being turned into a standing reserve and losing their 
capacity to be world disclosers. World disclosers [22] is a term used by Heidegger 
to describe how things in the world become meaningful through everyday human 
practices. Equipment has a purpose and when someone uses the equipment for the 
given purpose, it gives an identity to that person [23]. Hence peoples’ identities and 
sense of self are tied to things and practices. 


16.3.2 Focal Things and Focal Practices 


This connects to the concept of focal things and focal practices [15, 16]. The term 
derives from the word “focus”, which formerly referred a hearth or fireplace, which 
Borgmann used to illustrate the concept [15]. In the past, a fire was the focal point in 
a home, and people were brought together around it. It was the focus around which 
families gathered, shared news, told stories, played games and connected with one 
another. Maintaining the fire took effort, exertion and learned skills, from cutting 
and collecting the wood, to cleaning the hearth and setting the fire. The fire and 
its activities create sensory experiences, from the smells of the burning wood to 
the feeling of creeping warmth around a room or muscles fatigued from using an 
ax to split logs. Focal things like fires form a point of convergence for family and 
community life, and a social world revolves around them. The world is increasingly 
revealed, as the focal practice is habituated. 
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This is contrasted with technologies which require minimal effort, skill and 
engagement [15, 16]. Switching on a central heating system does not provide the 
same level of engagement and heat distributed through radiators allows a family to 
spread out across different rooms. Central heating gives the liberation of privacy and 
instant heat, at a cost. 


16.3.3 Commoditization 


Feenberg [4, 5] suggests that there are ways to challenge the tendency toward techno- 
logical de-worlding that results in wholesale commoditization. Rather than stripping 
away to reduce something to a raw material (termed Primary Instrumentalization), 
he advocates incorporation of the social into the technical so that technologies reveal 
more than commodity. Secondary Instrumentalization integrates the social environ- 
ment, through systemizations that disclose connections to the world, and valuative 
mediations that provide space for societal values to inform designs. 

In pre-industrial times when things were made in the place where they were used, 
the production networks were short and local cultural values and priorities were 
integrated and carried through production. In more recent times, technical networks 
involved in making digital devices are longer, often spanning the globe, and so the 
societal and cultural meanings and values that ground things and give them meaning 
beyond that of commodity may be lost along the way [5]. That said, longer production 
networks potentially involve more people and present increased opportunities for 
social interventions [5]. 

Feenberg argues that more voices are needed in the design process to represent a 
greater range of values and counter the control exerted through technical systems, a 
concept he called democratic rationality [4, 5]. From this perspective, technologies 
need to be open enough that they can be shaped or reconfigured by users to reflect 
their priorities. In this way the social is inevitably part of the design. 


16.4 Using the Critical Lens to Establish Design Guidelines 


Turning theory into practical guides to shape design is subjective because different 
designers will interpret and emphasize the theory in different ways. This subsection 
presents some selected design guides from the author’s interpretation of the theory 
above. 

Technological devices require little effort and skill so designs should be effortful 
and encourage skill development. 

Traditionally technology designers have determined what a design is and how it 
is used. This potentially reinforces power inequalities and excludes the input of users 
who will be affected by design choices. So, technologies should be flexible enough 
for wider society and users to influence and reshape designs. 
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Technologies slip into the background offering easy access to commodity which 
means there is no pause for interruption and reflection, so we lose sight of things, 
their world and ourselves. Consequently, designs or design process should create 
pauses and interruptions that nudge people into noticing things afresh and attending 
to them with more intent. 

Digital technologies don’t require deep engagement or connection to community 
or place so digital technologies should support focal practices and integrate social 
dimensions. 

Technologies have been criticized for creating distance and abstraction. To counter 
this, designs should be rooted in context by, for example, medium, materials, pro- 
cesses and values. In order for a design to connect to its world, production networks 
need to be short to stay close to local culture and values. 

Sensory stimuli connect people to the world of a thing. Smell, touch and taste are 
particularly associated with the immediate body space surrounding a person, so they 
are associated with immediacy and intimate connection. In contrast, visual stimuli 
are believed to create a sense of distance, between the seer and the seen [24]. Hence 
designs should encourage more intimate experiences and activate multiple sensory 
receptors. 


16.5 Artifacts 


These and other criteria informed the design of a series of objects for personal use 
and for interpretation of a public garden, including the Nature Meditation Egg, the 
Rhubaphone and Audio Apples (14, 25-28]. 

The Nature Meditation Egg (Fig. 16.1) was the first artifact made within the 
initial exploration and prototyping phase. It was designed for personal use by the 
lead author to explore the relationship between theory and practice in her role as 
designer—researcher. The Nature Meditation Egg was designed to encourage its user 
to notice the natural world. This involved the user audio recording the non-human 


Fig. 16.1 Nature Meditation Egg. Image copyright © J. Lindley 2015 and reproduced by permission 
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world on a walk in their local area. During meditation, the user would hold an egg- 
shaped wooden object which contained a sensor, microprocessor and audio player. 
When held with focus for a period of time, the audio from the walk would begin to 
play. The aims were to stimulate memories of the outdoor experience with the audio, 
and through the act of recording and listening draw attention to the natural world at 
a fine level of detail. The wooden egg was turned from local wood by a local joiner, 
though the digital components were manufactured elsewhere. 

The Rhubaphone and Audio Apples were made as digital hybrid interpretation 
for a public walled kitchen garden, to encourage people to notice it in new ways 
and support connection to place. The Rhubaphone (Fig. 16.2) was created to bring 
to attention the individual cultivars in the National Rhubarb Collection, grown in 
the garden. The Rhubaphone held stems of rhubarb side by side so that differences 
in size and color were more obvious. Holding a stem activated an audio recording 
about that variety, its history and cultural significance made by the Head Gardener. 
The smell of the cut rhubarb and touch of the stem added extra sensory layers to the 
experience. 

The Audio Apples took people into an Apple orchard, that held part of the Regional 
apple tree collection, but the design also spoke to the past and present of the garden, 
through audio stories, diary entries, audio recollections and poems recorded by gar- 
deners and volunteers (Fig. 16.3). The wooden Audio Apples hung from branches 
encouraging visitors to step off the path to reach and pluck them from the tree. Once 
plucked the different voices of gardeners and volunteers “spoke” to the visitors, 
through a variety of different kinds of recordings, intended to light a spark in the 
listener. 

The Rhubaphone and Audio Apples used wood from the site and the apples were 
turned by a member of the team. The heritage of the garden was incorporated through 
motifs that referenced the garden’s past. The design process was shaped to fit the 
working patterns in the garden and seasonal shifts within the garden. The iterative 
design process involved feedback and reflections from the Gardens team, and later in 
the collaboration, a volunteer learned to maintain the digital interpretation artifacts 


Fig. 16.2 The Rhubaphone in use. (Left) Image copyright © A. Johnson 2015 and reproduced by 
permission. Rhubarb cultivars. (Right) Image copyright © L. Edwards 2019 
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Fig. 16.3 Audio Apples in use. (Left) Audio Apples in the orchard. (Right) Image copyright © L. 
Edwards 2015 


and ultimately began to develop content and new ideas. Hence ownership over the 
artifacts started to shift from researcher—designer to the Gardens team. The software 
used was open source so that code wasn’t locked down and could be adjusted or 
completely rewritten. Content was stored on SD cards that were familiar to people 
from everyday use in cameras and phones. This meant people working in the garden 
could control content with only limited technical expertise. Where possible compo- 
nents were used that could be replaced to extend the life of the design or uncoupled 
by the team to create new designs. However, through the iterative design process 
it was found that pre-made boards incorporating microprocessors and sensors were 
more robust and easier for laypeople to manage. The design and development of 
these interpretation artifacts is expanded upon in Edwards [14]. 


16.6 Reflections on the Digital Nature Hybrid Designs 


Responses to the interpretation and its impact on visitors, volunteers and staff were 
collected through interviews, field notes and observations. The lead author (in her role 
as designer—researcher) made notes reflecting on design iterations as part of a research 
through design [29, 30] approach. She participated in the life of the garden and 
documented and reflected on experiences, conversation and observations in relation 
to the underpinning theory and practice in notes, audio recording and images made on 
site, or after visits. The effectiveness of the designs, in conforming to design criteria 
was evaluated with reference to these reflections as well as theory-driven thematic 
analysis of interviews with the Gardens team and visitors. 

The digital interpretation was positively received. Many visitors commented that 
they had not previously realized there was more than one variety of rhubarb or they 
hadn’t thought about rhubarb varieties. One of the gardeners said that the process of 
creating the Audio Apples helped the team realize the value of their stories, and also 
let them see how their colleagues experienced the garden. Both Audio Apples and 
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Rhubaphone prompted a pause in people’s passage through the garden, where they 
stopped and attended to the installations. 

The multisensory digital artifacts used touch, smell and sound to encourage a sense 
of intimacy with the artifact and place. The design criteria resulted in a particular 
aesthetic, that emphasized materials from the site. It produced technologies rooted 
in place and not easily used elsewhere. For example, the Rhubaphone incorporated 
heritage varieties only grown in the garden. Once cut, the rhubarb would begin to 
wilt and over time it would be unusable. This meant the Rhubaphone could only be 
used properly within the range of the garden, a characteristic that prioritizes the local 
and particular. 

Overall, the designs cohered with the design lens and had strong contextual rel- 
evance. The design processes were a product of place, fitting daily and seasonal 
patterns in the garden, and respecting the routines and practices of the team. At some 
points during the collaboration, design and maintenance activities were carried out 
in a shed in the garden in order to stay connected to the place and enable ongoing 
input from gardeners and volunteers. This was a response to Feenberg’s comments 
about the benefits of short production networks [5]. 

Material, software and technological choices were made with the gardeners and 
volunteers in mind. The question of what would be compatible with their working 
practices and what they might be able to use independently was kept in the fore- 
ground when making design decisions. This enabled a transition toward designs that 
were initiated by the team, though the team members who managed the digital inter- 
pretation artifacts felt more time was needed to build skills to gain more control of 
the software. The fact the designs were open meant their content could be driven 
by the team from the ground up, so that their priorities were centered. Stories and 
content could emerge from the garden. 

The interfaces and interactions were described as “fun” and “novel” and they 
interrupted users, as intended, triggering memories and prompting questions. The 
use of “actual rhubarb” brought the Rhubaphone to life. 


16.6.1 Challenges Arising Through Use of the Critical Lens 


Although the unfamiliarity of the interfaces caused people to pay attention, it also 
created barriers to use. The Audio Apples definitely didn’t slip easily and effort- 
lessly into the background and this was off-putting for some visitors and therefore 
potentially problematic for the organization. 

The technologies were designed to reveal the world of the garden, its values 
socially, historically and culturally but there were tensions as the interactive instal- 
lations could be critiqued as turning the things in the garden into commodities to be 
consumed by visitors. This conflicts with the aims of the design lens. 

The digital interpretation artifacts responded to many of the criteria drawn from 
the theoretical lens, but they didn’t achieve some of the objectives in relation to 
nurturing connection to place. The digital interactions created a pause, but often 
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this was fleeting and superficial and didn’t require enough active engagement from 
participants to really prompt reflection on place. They didn’t offer many opportunities 
for the kinds of extended, embodied engagements that create sensory surprises and 
experiences which change relationship to place and build emotional connection. 


16.6.2 Response to the Challenges Arising Through Use 
of the Critical Lens 


The Nature Meditation Egg was designed by its user (the lead author) and the process 
of designing, creating and using the egg added to a richer understanding of place. 
The relationship to place was stimulated by the act of design. The process required 
effort, skill acquisition and significant time spent in a natural environment, which 
are some of the characteristics of focal practices. 

These insights from the Nature Meditation Egg about how the creative process 
prompted reflection and seeded rich emotional relationships to the land influenced 
the next phase of work. The realization that the act of designing the artifacts unlocked 
understanding of place resonated with Heidegger’s writings about the nature of 
humans as world disclosers, who illuminate worlds through practices and make vis- 
ible the conditions on which things in the world depend. Making artifacts or stories 
in the garden required the creator to give time to the process and attend to the place. 
It involved sensory and experiential engagements that prompted emotional connec- 
tions and a relationship to place. It isn’t possible to control placemaking because 
relationships to place are individual and intangible but there is an argument that some 
conditions are conducive to nurturing connection. Active engagement appeared to 
play a significant role, and this insight echoes findings from earlier environmental 
education projects like Ambient Wood [31]. 

Reflecting on the initial Digital Nature Hybrid artifacts drove a new research phase 
about how digital technologies might support active engagement and exploration 
toward greater awareness of and connection to non-human nature and environments. 
This led to the inception of the Digital Naturalist program, which began in the 
public garden where the original work took place but has subsequently been further 
developed in a range of settings. 


16.7 Digital Naturalists 


The Digital Naturalist program comprises a series of workshops that combine arts 
activities, with digital making, computer programming, nature walks and fieldwork 
activities to reveal new perspectives on a place and rekindle naturalist skills. Cur- 
rently, there are two workshops in the Digital Naturalist program, Digital Boggarts 
and The Lost Sounds. The program builds on the insights gained from the Digital 


306 L. Edwards et al. 


Nature Hybrid artifacts. The pilot project (Digital Boggarts) which began in 2016 
extended the collaborative work with the National Trust gardeners and with writ- 
ers who had developed content for the Audio Apples. The second project (The Lost 
Sounds) has run in multiple locations across the North West and Midlands with a 
variety of audiences. 


16.7.1 Background 


The number of amateur naturalists has declined with successive generations [32] 
and children spend less time playing outside ranging across smaller territories than 
in previous generations [33]. Consequently, less than a quarter of children regularly 
use their “local patch of nature”, which is half the number in their parents’ genera- 
tion [34]. The Digital Naturalists workshops were designed to explore how digital 
technologies might be used to stimulate more sustained and effortful interactions in 
natural environments, with particular attention to sensory awareness. The workshops 
slow people down in order that they spend time in a place, while the activities aim 
to tune people into noticing details. The nature walks are led by local experts from 
environmental organizations in order to raise awareness of local groups. Participants 
may be more likely to get involved with these groups, as a member or volunteer, if 
they have made personal contact with someone in the organization. 

In the program, digital technologies are used to help people experience the natural 
world afresh by revealing things that may be otherwise hidden and by supporting 
the acquisition of traditional skills that bring a new perspective on the natural world. 
The aim is to inspire a new generation of amateur naturalists and citizens who have 
increased awareness of their local environments. 

Thinking through making is used to focus attention, consolidate knowledge and 
reflect on a place. The modular structure enables the workshops to be adapted for 
the specific contexts and audiences. Versions of the workshops have been created for 
formal and informal education and public engagement settings. 

As the workshops have progressed, the importance of sharing the knowledge 
and products resulting from the workshop, through an exhibition, has become more 
apparent. The exhibition extends the reach of the program to the wider community 
beyond workshop attendees and also presents opportunities for workshop participants 
to act as/or be experts. 


16.7.2 Digital Boggarts 


Boggarts are folkloric creatures, like hobgoblins, that are often associated with a 
particular location, for example, the Farndale Hob from North Yorkshire [35]. In 
effect they are spirits of a place. Traditional tales about Boggarts can be found across 
the United Kingdom making them a useful conduit for talking about place, nature 


16 From Digital Nature Hybrids to Digital Naturalists ... 307 


and history across the country. The pilot project, Digital Boggarts, drew attention to 
microclimates and planting within a public garden, and the spirit or feel of the place 
experienced as a result of these factors (Fig. 16.4). 

The project/workshop offered an opportunity to explore a small part of the garden 
in detail and build an intimate personal relationship with it. This study of a garden 
at micro level threw into relief the contrasting character, (or spirit) of differing areas 
within the garden to make these places present to the workshop participant. The 
Digital Boggart was a manifestation of the kind of creature that might live in that 
microclimate, a response to the question “Who would live in a place like this?” 

At first a wilder setting might seem more suitable than a formal, managed public 
heritage garden, but many people in the UK have access to a private or communal 
garden or yard, and gardens are often the most immediate point of contact with 
non-human nature. The public garden was far larger than a domestic garden but the 
scale made it easier to identify different microclimates. One of the aims was that 
participants might start to notice the microclimates and the spirit of different areas 
within their own gardens or communal spaces. Another aim was to encourage new 
visitors to the public garden. 

Twelve children aged 9-10 from a primary school located close to the public 
garden took part in the 1.5-day workshop, some was delivered in school, while 
the major part was delivered in the garden. In school, prior to the garden visit, the 
children learned to program Micro:bit computers to sense temperature, humidity and 
direction. The day in the garden began with a guided walk and mapping activity, led 
by gardeners. This was followed by time exploring the garden with sensors to identify 
microclimates. 

Using information collected in the garden as inspiration, the children worked in 
groups with a writer to make a fictional story about a Boggart that lived in an area of 
the garden. They performed the story in the garden and made an audio recording. The 
children created Boggart creatures from clay, beans, seeds and materials found in 
the garden, so that the character was rooted in the garden. Each Boggart was given a 
digital dimension in the form of a Near Field Communication (NFC) tag embedded 


Fig. 16.4 Images from the Digital Boggarts workshop: sensing humidity, story making and a Digital 
Boggart incorporating Near Field Communication tag. Image copyright © A. Darby 2019 
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in its body. When the Boggarts were placed on top of upturned flower pots (that 
housed hidden NFC readers and speakers), the story created by the group played 
aloud. In effect, the children created a piece of artistic digital interpretation inspired 
by their experiences and exploration in the garden. Interpretation is commonly used 
within museums, heritage sites and cultural organizations as a kind of communication 
that happens in conjunction with direct, firsthand visitor experience. Traditionally, it 
presented the voice of the expert, in a range of forms but there has been a move toward 
a more collaborative approach to include more voices alongside that of the trained, 
experienced interpreter [36, 37]. The idea behind this is the inclusion of excluded 
histories and the recognition of multiple perspectives on place. Back at school the 
children made a display of their interactive Digital Boggarts to share stories of the 
garden with their wider community. 


16.7.3 The Lost Sounds 


The Lost Sounds encourages people to pay attention to the birds in a place through 
their calls and songs (Fig. 16.5). The name has a twofold meaning. It refers first 
to the increasing number of birds on the International Union for Conservation of 
Nature “Red List” of species at risk of extinction. In 2015, 67 of 244 species were 
on the Red List, compared with 52 species, 6 years before in 2009 [38, 39]. This 
includes birds such as the Curlew, Nightingale, Merlin and Mistle Thrush. As their 
numbers decline, their songs and calls are lost to our localities and over time they fade 
from consciousness. The Lost Sounds is as much about the lost skill of recognizing 
birdcalls and the songs that are “lost to us” (as the number of amateur naturalists 
decline) as it is about the loss of particular species. 


Fig. 16.5 The Lost Sounds interactive birdcall prints. (Left) Image copyright © L. Edwards 
2019. The Lost Sounds mini exhibition. (Right) Image copyright National Trust Images 2019 and 
reproduced by permission 
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The Lost Sounds [40] has been run in formal education settings at primary schools 
and a nursery and informal education settings including community festivals and 
events run in partnership with environmental organizations/charities. 

It was originally designed for children in Year 5 targeting Key Stage 2 learning 
outcomes in computing, art and design, design technology, music and science. It has 
since been adapted for children in Year 3 and 4, nursery children and family groups. 
It is also being reconfigured for after school Code Clubs as a way of blending natural 
history and environmental science into the program. We anticipate compatibility 
between the structure of Code Clubs and Digital Naturalist workshops that may 
inspire the development of additional workshops. 

The workshop has a similar structure to the Digital Boggarts, involving time 
spent outside, combined with arts and technology. Specifically, the project includes 
a guided nature walk, birdwatching, sound recording and editing, listening skills, 
printmaking, circuit building and computer programming with Scratch running on 
Raspberry Pi. As before, the aim is to build naturalist skills including careful obser- 
vation and listening. The full workshop takes 2 or 3 days but shortened one-day and 
drop-in versions of the workshop have been run in settings where the longer format 
is unfeasible. This necessitates a smaller range of activities, but the workshop always 
involves art with technology. 

Ideally the workshop begins with a guided walk led by a bird expert or expert in 
the local environment, but in extreme weather timings may shift. Participants watch 
for birds with binoculars and take turns with a shotgun microphone and digital audio 
recorder to listen and make recordings at intervals on the walk. The aim is to use the 
digital equipment to tune people into their environment and make them more aware 
of the birdcalls around them. The directional capability of shotgun microphones 
helps participants to isolate the calls of individual birds, but they don’t require the 
same degree of precision as parabolic microphones. Sight is the dominant sense for 
most humans [24, 41—43] while sound is omnidirectional and dynamic making it 
less precise and harder to locate without a visual cue. Locating sound can require 
practice and effort. Using the microphone gives the holder a temporary “superpower” 
of enhanced hearing that makes it easier to notice sounds that are initially unfamiliar. 
As one becomes able to identify individual calls, it is easier to pick them out without 
the microphone. 

After the walk participants draw one of the birds found in the area from pictures, 
sometimes a bird they have seen or heard. The aim is to look carefully and closely 
observe the detail of the bird. The drawings (etched into foam) are turned into prints, 
made with conductive ink. In shorter workshops, participants are taught to recognize 
selected calls and then pin their print to an interactive sound sculpture, which plays 
the call of the bird when the print is touched. 

In the full workshop, participants either edit their own recordings or the ones made 
available under a Creative Commons license. They make a circuit with a capacitance 
sensor attached to the input and output pins of a Raspberry Pi computer and learn to 
program it to play appropriate calls when prints are touched. 

After these workshops, the participants’ prints and edited audio are transferred to 
an installation comprising: Bare Conductive Touch Boards, speakers and fabric print 
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of a location visited during the workshop. The installation acts as a mini exhibition 
of the outputs of The Lost Sounds workshop and it encourages involvement from the 
wider community beyond workshop participants, (for example, parents and other 
pupils in a school, or National Trust visitors or local residents). The interactive 
soundscape installation has been designed in a form that is easy for external partners 
to manage. 


16.8 Reflections on Digital Naturalists 


Both strands of the Digital Naturalist program, Digital Boggarts and The Lost Sounds 
have been well received by participants, host organizations, teachers and parents in all 
settings. Focused time spent in place, and time spent reflecting on non-human nature 
is built into the design of the program, so this inevitably results in a more significant 
experience for participants than that created by digital nature interpretations alone. 

The combination of arts, computing and outdoor experiences is particularly potent 
because the digital technologies like the audio recorders enable the participant to 
experience the environment differently and reveal things that it would otherwise be 
harder to sense. The digital technologies don’t supplant the traditional noticing skills 
but awaken them, whether that is noticing sunny and shady parts of a garden or using 
directional microphones to pick out the sounds of individual birdcalls. The digital 
technologies are intended to encourage people to spend time outside in a focused 
way. This doesn’t have the value of habitual focal practices, but it does require effort 
while also contributing to skill acquisition that can change a person’s relationship to 
a place and potentially their relationship to technologies. 

Creative practices require different kinds of engagement. The drawing encourages 
the participant to look carefully and notice the difference and the printmaking asks 
people to think about the character and signs that communicate the essence of a 
bird species. To write the Boggart stories, people have to think imaginatively about 
the space which goes beyond recalling facts. The story outcomes may start from 
jumping off points in a world that is close to the reality of the garden and then shift 
into realms of fantasy, but both have an impact on the memory of the children and 
the connections they make to the garden. A translation occurs as the participants 
turn what has been shared with them into something that they own, and that process 
involves the investment of the participants. 

The layered activities provide different hooks for individuals. Some respond to 
the making activities, while others are drawn to computing, but the combination is 
intended to make each activity more appealing to a wider audience. The intention is 
to consider how this interdisciplinary approach affects perceptions, particularly of 
technology, computing and environment. 

One of the most significant outcomes has been the impact of the sound installation 
that creates a mini exhibition of project work. These have been left in schools or 
National Trust properties for days or weeks after a workshop to open the project out 
to wider audiences who did not participate directly in the workshops. In schools, 
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teachers and children talked about the pride of showing their creations to others 
and sharing the birdcalls they had learned. The fact that the fabric image was of 
the place where they had spent time outside birdwatching and recording also made 
it more meaningful. Showing the work to others in the community, for example, 
at Community Days can start conversations about birds in the area and how to get 
involved with projects to protect their habitat. 

The role of environmental and creative experts is also a key feature of the workshop 
design. Local environment experts reveal what is hidden or unnoticed and when this 
is paired well with technologies that also make the invisible visible (or audible), 
they complement each other powerfully. Creative experts like writers or printmakers 
introduce ways of seeing that may also unlock new dimensions of non-human worlds. 
These inspire the active engagements that build care for the non-human world. The 
creative activities create a space for conversations and questions about ecology and 
action. Beyond the participants, the workshops bring together organizations and 
individuals with different competencies who have an interest in the environment, 
and this has the potential to strengthen networks and seed collaborative action. 

The design criteria used at the outset emphasized that the designs should fit the 
place in which they will be situated. In the earlier Digital Nature Hybrid interpreta- 
tion, this included using materials from the site and designing artifacts with the local 
team. This presents a challenge for the Digital Naturalists workshops because they 
move from place to place. For example, the materials for the exhibition stand cannot 
come from the location unless the stand is continually remade. 

Nevertheless, the Digital Naturalist workshops have been designed to give promi- 
nence to place and to respect individual contexts. For example, the Digital Boggarts 
incorporate materials from the garden in which the Boggarts “live”. The fabric back- 
drop for the Lost Sounds is reprinted with a site-specific photo for new workshop 
locations. Working with experts from a site and planning the activities with partners 
who know the location well is another way of ensuring that care for the place is 
embedded, and that the use of digital technology is appropriate. The combination of 
arts, outdoor activities and computing is important because at different times each 
pulls place into focus. That means there is a tension when the shortened versions of 
the workshops are run because an element is lost that may weaken the connection 
to place. However, the shortened workshops have prompted bookings for the longer 
workshops. 

One of the ways that Digital Naturalists responds well to the original design lens is 
that the workshops encourage attention, endeavor and the development of skills. The 
commitment required in time and effort makes these engagements potentially more 
meaningful than the brief encounters with the Digital Nature Hybrid interpretation. 
In both cases, the participant’s intent and motivation will affect their experiences. 
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16.9 Conclusions 


Technology has been criticized because of its instrumentalizing tendency, and the 
implications of that tendency, for example, the increasing separation of humans from 
non-human nature that results in a lack of care that puts the environment under threat. 
It also includes the impact on humans’ sense of self and well-being. By responding 
to the criticism directed at technologies through design criteria that referenced the 
criticism directly, the projects aimed to refocus and reshape priorities and design 
processes. The simple act of setting the criteria was important for foregrounding the 
questions, “How are we using digital technologies?” and “What impact is this having 
on humans and non-humans?” 

One of the significant impacts of this lens was the cascade created by prioritizing 
individual places and rejecting the idea that “one size fits all.” Attending to place 
meant the design process had to respect local ways of working including the rhythms 
of the place, for example, seasonal and daily working patterns. This in turn influenced 
digital technology choices and other material decisions which ultimately influenced 
physical designs and sensory interactions. By attending to place within the design 
process, the designer—researcher (lead author) became more connected to the places 
in which she designed. This led to the insight that making in a place and in response 
to a place could foster a connection and a realization which drove the second phase 
of research. 

The initial Digital Nature Hybrid designs went some way to revealing places and 
showing non-human nature in a new light. It demonstrated how digital technologies 
could be rooted in context and culture and how they could amplify particular sensory 
stimuli to resonate with the sense of place. This in itself had value, for designers, 
organization and visitors. 

However, the engagements with the Digital Nature Hybrids were brief and too 
superficial to prompt significant change or connection, unless the people concerned 
were already primed for change or connection. Analyzing the limitations of the 
Digital Nature Hybrids with respect to the design criteria showed that effort, focus, 
skill and social connection that made focal things and practices meaningful were less 
present in the Rhubaphone and Audio Apples and so the question of how to build in 
these attributes became central to the second phase. 

The subsequent projects created deeper, richer encounters with natural environ- 
ments and showed the power of combining artistic activities with technological activ- 
ities. The activities required participants to use their technologically amplified senses 
and creative skills to attend to and reflect on the world. 

Using technologies to kindle traditional skills and practices shows one way that 
technologies might contribute to re-energizing the culture of amateur naturalists and 
nurture care for non-human nature. 
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I. Pioaru 


Abstract Virtual reality holography (VRH) is a new art form that synthesizes the 
qualities of traditional hand drawing with the unique features of holography and 
virtual reality (VR) art. As a new art-making tool, it offers the possibility to develop 
artistic ideas and concepts that could not be materialized with any other medium, 
whilst from a practical point of view it has the unique ability to showcase VR artworks 
external to VR space, without the need for a headset. Starting with a brief historical 
overview of holography art and VR art, this chapter presents an exploration of the 
VRH medium and the artworks which were generated. The various contexts in which 
these artworks are disseminated are presented, and the dialogues arising from the 
material specificity of VRH as a new medium are discussed. VRH is envisaged to 
extend the reach of audience engagement. 


Keywords Virtual reality holography - Digital holography - New media « Virtual 
reality art 


17.1 Introduction 


Virtual Reality (VR) art-making tools have become increasingly popular amongst 
artists. However, VR art is still far from ubiquitous, mainly because of the high cost 
of purchasing a VR system and the space required to run it. At the same time, VR 
artworks are fundamentally difficult to view or exhibit outside of VR, as they have a 
functionality that makes them impossible to reproduce in any other medium without 
losing some of their essential features. For example, 3D printing, due to its particular 
constraints, only allows for a relatively narrow range of models to be produced [1]. 
Online galleries which allow the viewer to explore VR projects using a standard 
monitor, flatten the three-dimensionality of VR artworks, as is the case with any 2D 
representation of a volume [2]. 
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Digital holography, despite its own limitations, can constitute good support for VR 
imagery, offering a high fidelity of representation and preserving the most important 
properties of such imagery. 

The advantage of bringing together these relatively new media is that it allows an 
artwork produced in VR to be displayed and viewed easily in either a public gallery 
or a private space, by a much larger number of people simultaneously, without the 
constraint of requiring viewers to wear a cumbersome headset in order to preserve 
the 3D appearance. 


17.2 Overview of Holography as an Artistic Medium 


In simple terms, a hologram is a photographic recording of a light field, on either film 
or glass, resulting in a seemingly 3D image which can be seen with the naked eye. 
Holography can be either analog, when it captures something which exists in reality, 
or digital, when it displays computer-generated objects or scenes. Both types of 
holograms require a special, dedicated illumination system in order to be viewed, in 
the absence of which they appear as dark, indistinct, flat surfaces. There are a number 
of techniques and optical illusions which are commonly and mistakenly referred to as 
‘holography’, such as Pepper’s Ghost or various stereoscopic and lenticular displays. 
Although holography practitioners will reject these as mere tricks, the general public 
is perhaps more familiar with such techniques than with holography itself, due mainly 
to the fact that they are often employed in funfairs, in theatre and in other on-stage 
acts. Arguably the most notorious use of Pepper’s Ghost was Tupac Shakur’s virtual 
appearance at Coachella Valley Music and Arts Festival, in 2012, alongside Dr. Dre 
and Snoop Dogg [3]. 

Although primarily a scientific field, holography has a significant potential for 
being used as an art medium and, as a consequence, the domain appeals equally to 
artists and scientists. Since its invention by Denis Gabor in the 1940s, holography has 
drawn the attention of a number of artists, amongst whom Salvador Dali is perhaps 
the best known [4]. Many of these artists have dedicated their career to the medium 
of holography, for example, Margaret Benyon [5] or August Muth [6], whilst others 
have tried to incorporate it in, or add it to, their existing practice, as in the case of 
Moysés Baumstein. 

Catalogues, photographic documentation and generally any type of literature on 
holography don’t really do justice to a medium that doesn’t lend itself easily to 2D 
reproduction. However, articles on holography written by practicing artists, curators 
or critics are useful for understanding how holography was, and is, perceived and 
experienced first-hand. 

Two volumes of the Leonardo art and science magazine dedicated to holography 
art [7] comprise numerous interviews and articles where holography art practitioners 
amongst whom Margaret Benyon, Andrew Pepper and Paula Dawson express their 
dissatisfaction with how the art world receives their work. 
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Perhaps the most striking concept that comes from these texts, and others written 
later, is that holography and the art establishment still don’t feel comfortable with one 
another. The medium of holography has encountered constant criticism and dismissal 
from the art establishment. The main points made against it being vulgarity of colours, 
awkwardness of display and lack of a proper critical and conceptual vocabulary. This 
is revealed by D. Tulla Lightfoot in a study titled “Contemporary Art World Bias in 
Regard to Display Holography’ [8], where the author interviews several curators 
and directors of art galleries in New York, seeking to understand their views on 
holography art and how they justify the constant rejection of this medium by the 
establishment. The main conclusion of the study is that major museums and galleries 
don’t necessarily dismiss the medium itself—they base their rejection on the status 
of the artist, claiming that, if an artist who has previously gained recognition by 
working with other media decided to take up holography, they, the art venues, would 
not hesitate to show their work. If this is the case, one might infer that holographers 
simply face the same problems as any other artist approaching a venue seeking 
representation; with the added difficulty of having chosen a medium that hasn’t 
really managed to prove its aesthetic or conceptual value, as yet. 

Nevertheless, there seems to be an ongoing struggle amongst holography practi- 
tioners to be accepted and valued or sometimes even considered by major art venues, 
in spite of long years of sustained efforts and obvious developments, both in the 
field of holography as a whole and within the individual practices of different artists. 
During a presentation at the International Symposium for Display Holography 2018, 
August Muth, one of the most successful artists working with holography today, 
confessed that when approaching a gallery to propose a collaboration, he simply 
steered away from using the word ‘hologram’ in reference to his artworks. This was 
to avoid the immediate negative reaction and consequent rejection which are usually 
triggered by the mere mention of this word. 

It must be acknowledged that the medium still has glaring deficiencies and limita- 
tions that make holographic artworks unconvincing and could be the reason for recur- 
ring negative responses. For example, common limitations mentioned are brightness 
and colour balance, the complexity of arranging good replay illumination, blurring 
in the rear of a scene, vertical non-uniformity of colour, digital pixelization and the 
lack of a good archival quality photographic recording material. 

Perhaps one of the reasons why many attempts at making holography art have 
failed, is that the practice of holography bears no clear line of demarcation for 
where the science ends and where the art begins or vice versa. In order for a holo- 
graphic product, and any new media artwork to be successful from both points of 
view—technical/scientific and artistic—this distinction needs to be acknowledged 
and embraced. 

Some of the most successful examples of holographic art come from practitioners 
who have understood the importance of limiting the role of the artist and that of the 
scientist in order to allow each of them to contribute optimally in this equation. 

A collaborative approach between artists and scientists is needed to enable the 
two fields to intersect in a successful way that leads to further progress. New media 
often come with the temptation to use their novelty and specificity as an objective 
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in itself. As a result, practitioners with a background in art are unable to produce 
artworks of satisfactory technical quality, whilst technicians or scientists with no 
artistic background, seduced by the visually enticing qualities of the media, produce 
trivial objects of no real artistic value. When working with a difficult medium such 
as holography, it is important that both the artists and the scientists understand the 
limits of their knowledge, and by using each other’s expertise and providing constant 
feedback, work together towards taking the technology to a level that allows it to be 
acknowledged by the establishment. 


17.3 Virtual Reality Art 


To clarify the terminology, VR art is a type of artistic content created directly inside 
the VR environment using an application designed specifically for the purpose of 
art-making, and not computer-generated content presented in VR. 

VR art-making applications such as Tilt Brush or Gravity Sketch are impressive 
tools for creating digital 3D imagery, offering great new possibilities for art and the 
creative industries. They stand out amongst other VR applications due to the fact that 
they allow users to generate virtual reality content, rather than simply consuming 
it. This unprecedented combination of painting and sculpture, not bound to physi- 
cal laws, with its lack of spatial limits and highly intuitive interface, offers a very 
satisfying immediacy that materializes gestures instantly. 

With its impressive variety of brushes often accompanied by special visual and 
sound effects, Tilt Brush was designed with the intention of offering the user not 
just art tool but a fully immersive VR experience, entertaining in and by itself. 
On the other hand, Gravity Sketch, with its focus on creating geometry in the VR 
environment, has proved a real game changer by providing a level of editability 
and precision much superior to Tilt Brush, features which appeal greatly to design- 
oriented creatives. More recently, the team behind the VR animation tool Tvori 
aim to create a powerful, immersive and intuitive tool for crafting visual stories, 
which anyone can use, regardless of their previous experience with either 2D or 3D 
animation [9]. 

Currently an increasing number of artists have expressed their interest in this 
medium and have started adding VR art-making to their ‘tool box’. However, because 
VR art is still in its infancy, and also because of the difficulties associated with 
showcasing these productions outside of VR space, the works realized in this medium 
are only accessible to the wider public via online 2D galleries [2]. 


17.4 The Creation of a New Artistic Medium 


The most important achievement of holographic imagery and what truly differentiates 
it from other media is its capacity to visually expand space without resorting to the 


17 Virtual Reality Holography—A New Art Form 321 


traditionally accepted convention of treating a 2D surface as an interface for 3D 
content. In seeking to redefine maximalism as a type of art which uses a minimum 
amount of space to deliver a maximum amount of information or content, holography 
becomes a good example of an efficient use of the ‘space of art’, in the sense that it 
employs a flat surface to display a volume, rather than the actual 3D space usually 
taken up by a physical object or scene. 


17.4.1 ‘Van Gogh by Pioaru’ 


By using this unique quality of holography in conjunction with VR art, there is an 
opportunity to not only bridge the gap between the virtual and the real, but also to 
develop a new form of art that could be unique in itself. 

Therefore, a digital hologram was created from a VR artwork. The project was 
implemented using Google Tilt Brush and an HTC Vive headset and, since it was a 
reinterpretation of a Van Gogh self-portrait, it was entitled ‘Van Gogh by Pioaru’. 

In order to transfer a VR project onto a holographic format (and, to my knowledge, 
the first time anyone had attempted such a transfer), attention was focused principally 
on the technological aspects of the process, and the choice of subject was a secondary 
consideration. 

Part of the process of transferring the VR project to holography consisted of 
using the FBX file format for exporting the scene. However, due to various software 
incompatibilities, only the volumes were imported from TB into 3ds Max, whilst 
the colour and texture information was lost. This meant that the scene had to be 
recolored in 3ds Max. In addition, when imported, the brush strokes were reduced 
to plain, ribbon-like meshes. All this resulted in significant differences between the 
original VR scene and the final hologram, which was printed using a commercial 
digital holographic printer from the data set (Fig. 17.1). 


17.4.2 ‘Spectral Figures’ 


In spite of all technical difficulties, Van Gogh by Pioaru was a promising first step. 
The artwork was well received at a public demonstration. However, there were some 
constraints and these were addressed in a follow-on project, “Spectral Figures’ which 
made adjustments to the limitations of the transfer process. This consisted of a series 
of black and white portraits of artists and philosophers (Fig. 17.2). These were drawn 
in black and white in order to reduce the risks associated with recoloring the scene 
(described in the previous section). In order to maintain a degree of similarity to the 
models, several photographic portraits of the subjects were imported into Tilt Brush. 
These were sourced online and taken from different angles, and used as reference 
images for the artworks, with the objective of building into the models the correct 
volumetric structure. Thin black strokes were used set against a white background in 
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Fig. 17.1 A snapshot taken in Tilt Brush (left) and a photograph of the final hologram (right). 
Copyright © I. Pioaru 2019 


Fig. 17.2 ‘Spectral Figures’ series—a collage of snapshots taken in Tilt Brush. Copyright © I. 
Pioaru 2019 


order to simulate the characteristics of drawing on paper. As a consequence, possibly 
the most striking aspect of the resulting images is that, although effectively they are 
sculptures, every viewing angle gives the illusion of a 2D line drawing [10]. 

This decision brought the project a step closer to a preferred artistic media, i.e. 
drawing, and particularly to ‘sculptural drawing’. This is a concept which has been 
approached in several recent projects and it refers to expanding the characteristics 
of drawing to create sculptural objects, as part of an exploration of the boundaries 
between 2D and 3D imagery. Figure 17.3 is a visualization of a project which best 
illustrates this concept: Insectarium, 2012—a sculptural drawing installation—fol- 
lowed by two close-up photographs (Figs. 17.4 and 17.5) of one of the boxes which 
make up the installation. 
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Fig. 17.3 ‘Insectarium’ by Ioana Pioaru—sculptural drawing installation, 2016. Copyright © I. 
Pioaru 2019 


Figs. 17.4and 17.5 ‘Insectarium’—sculptural drawing installation, close-up. Copyright © I. Pioaru 
2019 
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The pipeline for creating a hologram from a VR project is relatively straightfor- 
ward. In order to generate the image data required by the digital holographic printer 
for a single-parallax hologram, the VR project needs to be exported as a Filmbox 
(FBX) file—a feature initially implemented into Tilt Brush to allow artists to share 
their creations online. This is a format that makes it possible for digital content to 
be manipulated across a variety of digital creation software. The incompatibilities 
between Tilt Brush and 3ds Max were addressed by simplifying the scene and using 
a minimum of colours—effectively just black and white. This reduced the time of 
post-processing significantly, the main remaining task being the setup of the cam- 
era for image rendering. This is achieved by programming an automated camera in 
3ds Max, a process described by David Brotherton-Ratcliffe and Hans Bjelkhagen 
in their book ‘Ultra-realistic Imaging’ [11]. This method produces the required per- 
spective views that can be pixel swapped to generate the data necessary for the digital 
holographic printer. 

The data was sent to Geola for printing on their digital holographic printer and 
a series of 30 cm x 40 cm colour reflection holograms was produced using a sil- 
ver halide emulsion. The holograms, when properly illuminated, produce a good 
impression of the original VR subject (Fig. 17.6). 

Digital holograms may generally either be ‘Full-Parallax’ or ‘Single-Parallax’. 
‘Single-Parallax’ holograms are also sometimes referred to as ‘Horizontal-Parallax- 
Only’ holograms. In general, the ‘Full-Parallax’ hologram is to be preferred as it 
presents the most general and most faithful 3D experience of the original light field 
to the viewer. The viewer will be able to see both to the left and to the right of an 
image, as well as properly perceive the image from above and below. The viewer 
will also be able to approach the image and observe it from close-up; in brief, a 


Fig. 17.6 ‘Spectral figures #4’, single-parallax hologram (three photographic views). Copyright © 
I. Pioaru 2019 
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full-parallax hologram will relatively faithfully reproduce the original 3D light field 
from the active side of the hologram. 

Single-Parallax or Horizontal-Only Parallax holograms remove the vertical par- 
allax from the light field. As the viewer moves their head up and down, the 3D image 
will appear to simply tilt up and down. Since human eyes are horizontally separated, 
this type of hologram still looks very realistic to most people. The tilting of the image 
is fairly well accepted by the viewer and interpreted as a natural movement of the 
image itself. The largest drawback with single-parallax holograms is that the viewer 
cannot approach too closely to the hologram without image distortion becoming 
excessive. 

There are several advantages of the Single-Parallax hologram. The first is the 
digital image data is usually some hundreds of times smaller than the data required 
for full parallax holograms. This means that whilst single-parallax holograms can 
be calculated on a normal laptop or PC, full-parallax holograms often need either 
a reasonably large network of PCs or a small supercomputer. The second principal 
advantage of the single-parallax hologram is that it can be illuminated much more 
easily than the full-parallax hologram leading to brighter images and deeper in-focus 
scenes. 

Nevertheless, full-parallax holograms stand out as being far superior in that they 
convey the 3D reality of the drawings in a much more efficient and successful manner. 
The missing vertical parallax is in the single-parallax holograms acts to destroy the 
illusion of the image integrity that these sculptural drawings seek to embody. The 
single-parallax holograms are brighter and they can be illuminated by arrays of 
lights, making them potentially much brighter than the double parallax holograms. 
However, this does not really compensate for the critical loss of the vertical parallax. 

Fully un-apertured double-parallax holograms, giving the maximum field of view, 
is useful if the hologram is to give the best illusion of reality, printed on Silver Halide 
material. However, this may inevitably lead to a relatively poor brightness at replay, 
even when powerful LED lights are used for illumination. Improving this brightness 
is an important ‘next step’. Potentially, photopolymer could be used instead of Silver 
Halide, but a few companies are currently offering this choice. Other solutions would 
be to use more powerful laser diodes for the illumination. The best possible result 
would be expected if dichromated gelatin glass plates were to be used in the printer. 


17.4.3 Tilgate Forest 


The starting point of this project was a homonymous ink drawing first created in 
2018. This was an opportunity to explore a theme that was different from previous 
VRH artworks, which had been exclusively portraits. In addition to this, since the 
landscape is rather underrepresented in holography this choice of theme seemed 
likely to produce an interesting result. 

A photograph of the original was imported into Tilt Brush to use as a reference 
image (Fig. 17.7). 
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Fig. 17.7 Tilgate Forest by Ioana Pioaru, original ink on paper drawing, 2018. Copyright © I. 
Pioaru 2019 


Several factors identified in the initial drawing pointed to the possibility of struc- 
turing a 3d model of this scene as a diorama: the way in which the elements of 
the composition are clearly individualized, grouped and layered at varying depths; 
the relative difference in scale of these elements; and the amount of graphic detail 
used for rendering each layer. Thus, whilst drawing the scene in VR, the composi- 
tion was distorted by making objects in the distance much smaller and closer to the 
main viewpoint than they would be in an accurate scale model of the real landscape. 
Figure 17.8 shows the scene from the main viewpoint and Fig. 17.9 shows what it is 
as seen from above. 

The compressed depth also offered several advantages holographically, primarily 
that of limiting the angular resolution required to resolve the rearmost parts of the 
scene. This is a particularly important consideration in light of the fine brush strokes 
employed. Another advantage is geometric: due to the ‘window’ nature of a hologram, 
scene components nearer the glass are viewable from a wider angle of view before 
being obscured by the edge of the hologram. Thus, compressing the scene makes the 
compositional layout more robust to viewer movement. 

The use of the brushes roughly followed the same method employed for previous 
projects, in that the sense that the brush used for the white areas had to be different 
from that used for the black areas. 

The resulting ‘sculptural drawing’ was close enough to the original ink drawing 
and quite convincing as a 3D depiction of a landscape, in terms of the parallax effect. 
However, pure black and white images such as ‘Spectral Figures’ and “Tilgate Forest’ 
represent particularly challenging subjects for holography, straining its capabilities 
with respect to brightness and colour balance. Additionally, whilst the maximum 
angular resolution of digital holography itself is state of the art (thousands of views), 
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Fig. 17.8 ‘Tilgate Forest’: snapshot taken in Tilt Brush. Copyright © I. Pioaru 2019 
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Fig. 17.9 ‘Tilgate Forest’: scene viewed from above. Copyright © I. Pioaru 2019 


current replay illumination techniques reduce this by an order of magnitude. As 
a result, although the depth compression used in “Tilgate Forest’ was an effective 
illusion in its own right, it was insufficient to fully mitigate the blurring imposed on the 
background of the scene. This somewhat compromised the pen-drawing appearance, 
the very fine brush strokes seemingly blending together to create a charcoal-like 
effect (Fig. 17.10). 
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Fig. 17.10 ‘Tilgate Forest’: photograph of the hologram. Copyright © I. Pioaru 2019 


17.4.4 Heterotopia 


The project titled ‘Heterotopia’ is also based on an earlier work, a two-colour reduc- 
tion linocut print, part of a larger series (Fig. 17.11). This combined architectural 
structures and pseudo-realistic machinery, in the ‘techno pop’ style characteristic of 
recent art practice, where the stark, rigid appearance of the machines and buildings 
contrasts with the rawness and vividness of the colours and the cartoon style. Grav- 
ity Sketch seemed much more appropriate a tool, as it offers the features necessary 
for achieving this type of imagery: the possibility to generate and edit basic shapes 
and volumes, a certain degree of precision and perhaps most importantly, a grid and 
world axes/coordinates system that the objects can be aligned to. 

As the original artwork was a flat, frontal representation of the architec- 
tural/‘machinic’ structure, adding depth was the first obvious step necessary in the 
creation of the 3D model of the subject. Consequently, a roof-type surface and a 
lateral wall were constructed and filled with graphic detail, so that in the hologram, 
the roof—an element that is not conceptually essential to the composition—would 
be revealed and concealed by the viewer moving their head up and down. The added 
lateral wall was intended as a ‘transition area’ combining architectural features with 
‘machinic’ details (Fig. 17.12). 

The post-processing necessary to prepare the data for the holographic printer was 
different and much more complex than those required by previous projects, and was 
therefore done in collaboration with Tal Stokes from Geola Technologies, whose 
knowledge and expertise were essential (Fig. 17.13). 
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Fig. 17.11 ‘Heterotopia’ series: first print (left) and colour permutations digital mock-up (right). 
Copyright © I. Pioaru 2019 


17.5 Presenting Virtual Reality Holograms to the Public 


The ‘Van Gogh by Pioaru’ hologram was first presented in 2017 at the Cyberworlds 
International Conference in Chester. During the 2018 Focus Wales Festival, it was 
on display in one of the exhibitions organized at the Ty Pawb Gallery in Wrexham. 

The ‘Spectral Figures’ VRH series was presented for the first time at the 11th 
International Symposium for Display Holography which took place in Portugal in 
2018. On the same occasion, it was part of the holography exhibition held at the 
Museum of the City of Aveiro, titled ‘Art in Holography: Light, Space and Time’ 
[12]. In 2019, during the Artists’ Open House event held within the Dulwich Festival, 
‘Spectral Figures #1’ was exhibited together with some recent artworks made in other 
media. 
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Fig. 17.12 ‘Heterotopia’: collage of three views of the Gravity Sketch model. Copyright © I. Pioaru 
2019 


Fig. 17.13 “Heterotopia’: 
photograph of the hologram. 
Copyright © I. Pioaru 2019 


‘Tilgate Forest’ and ‘Heterotopia’ were presented at the SPIE Photonics West 
conference which took place in San Francisco in February 2019, together with one 
of the pieces from the ‘Spectral Figures’ series. 

On each occasion, the work was well received both by the general public and by 
artists and scientists attending the events and was deemed highly innovative, partic- 


ularly the concept and method of combining traditional drawing with the medium of 
digital holography. 
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In August 2019, a selection of the VR holograms will be included in a group exhi- 
bition which will look at the artistic drive being immersed by the image, organized 
at the Centre for the Holographic Arts in New York. 


17.6 Discussion 


For artists and researchers interested in ultra-realistic imagery, it may be important 
to understand how these relatively new art-making media, holography and virtual 
reality sculptural drawing, which share the ephemeral nature of their content, can be 
employed together and how they can highlight and enhance each other’s potential. 

Major companies developing VR technology are striving to make their devices and 
products more affordable, lighter, wireless, more intuitive and to offer the possibility 
to swap between AR and VR. This will inevitably impact on how artists make, 
experience, and understand art. On the other hand, it may not be necessary to find 
ways of ‘exporting’ productions of VR to other platforms, simply because the best 
way to experience them will be inside VR. Nevertheless, due to the challenges and 
difficulties associated with such ambitious projects, VR technology is still far from 
ubiquitous. 

In this transition period, it is therefore relevant to examine how the intrinsic gap 
between the virtual and the real can be bridged. As a stage in developing Tilt Brush, 
Google has recently initiated an Artist in Residence programme [13], emphasizing the 
importance of an active collaboration between artists and scientists in understanding 
what steps are necessary to take VR art-making to the next level. Because of this, 
many artists are expected to use VR as an instrument to develop their practice, and 
this highlights the need to find ways to exhibit their creations. 

VR holography is currently dependent on commercial digital holographic printing 
services. These services take the processed VR data set and generate the physical 
hologram. Perhaps the main problem for the practicing VRH artist today is that 
commercial holographic printers usually print on plastic film. To achieve the flatness 
required for the display of the artwork, this film must be laminated onto either a 
plastic or glass sheet using an optical glue. Unfortunately, even if the visual result 
is usually excellent using this technique, the process itself is associated with poor 
archival properties. This is a significant problem for the artist and indeed for the wider 
art collecting industry. The situation is slowly changing in this respect and digital 
glass hologram printers are starting to appear. In the EU, Geola, in addition to offering 
large holographic prints (up to 1.5 m x 1 m) on film now offers a service for printing 
some types of smaller glass holograms (up to 15 cm x 15 cm) on photoresist. The 
Centre for Ultrarealistic Imaging at Glyndwr University in Wales is also planning 
on recommissioning its large glass printer (capable of printing up to 1.2 m x 1.2 m) 
later this year. 

Another problem with commercial holographic printing is the availability of small 
hogel sizes. Currently, Geola offers a 0.8 mm hogel with its film printers although its 
new glass service can in principle goes down to 100 microns. ‘Hogelization’ or the 
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visual pixelation caused by larger hogel sizes is a distracting feature in the context 
of VR holographic art. It is a particular problem with full-parallax holograms where 
the nature of the image invites the viewer to view the artwork from all distances 
including from very close proximities. Reducing the hogel size by a factor of two 
does however increase print time by a factor of 4. And with a 30 x 40cm x 0.8 mm 
hogel hologram usually printing in around 3-3 h, commercial printers are not keen 
to reduce the hogel size dramatically. Nevertheless, technology is again coming to 
the rescue here with the latest generation of digital printers now forecast to offer 
printing speeds between 120 and 180 hogels per second—a significant improvement 
over typical current rates of 30 hogels per second. 

A further problem encountered with VR full-parallax holography is image bright- 
ness. This is usually related to the preferred use of Silver Halide emulsions in the 
hologram printing industry. Sculptural VR drawings can be particularly susceptible 
to this brightness problem if there is too much white (or coloured) background and 
too few black drawing lines. 

A way around the brightness issue is to use the technique of drawing with a white 
pen and a black background. This effectively routes the light energy available from 
the hologram into the drawing lines. Because the drawing lines usually occupy a 
much smaller solid angle in total than the background, the relative brightness of the 
lines can be much higher. Another effect of this technique is that when the hologram 
is switched on, the only thing which changes is the image lines appearing in front 
of a black background. This is in contrast to the case of drawing with a ‘black’ pen. 
In this case, when the hologram is switched on, the background changes from black 
to white. A further advantage of this technique is that hogelization should be less 
perceptible through the bright pen lines when compared to their perceptibility through 
a white background. ‘White’ is also the hardest colour to achieve in holography and 
large areas of white tend to often show up discoloration. Discoloration should be 
minimized by both the geometry and brightness of the lines. 


17.7 Conclusions 


The primary conclusion of this paper is that digital full-parallax full-colour hologra- 
phy provides an appropriate and useful means of visualizing 3D art created in VR. 
Whilst digital holography is not without its problems, it is currently the only medium 
that affords simultaneous group viewing of 3D virtual works without equipment such 
as glasses or head-mounted devices. This is an important consideration in the con- 
text of an art gallery. The artist should therefore be encouraged to consider its flaws 
as simply an inherent constraint of the medium and understand how to work with 
the medium and how to use it to its best effect. In this regard, VR holography is 
no different from any other medium. Perhaps the principal difference is the highly 
technical nature of holography and the burden that this places on the artist. 

The next step is to present these VRH artworks in the context of an art gallery. Here 
they could be brought to the attention of a wider audience and also art experts who 
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could assess their relevance within the landscape of contemporary art. Hopefully, the 
dissemination of VRH artworks would then result in more art practitioners deciding 
to work with it and finding it stimulating, useful and inspiring. 
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Chapter 18 A) 
In Darwin’s Garden: An Evolutionary geit 
Exploration of Augmented Reality 

in Practice 


Alan Summers 


Abstract This chapter discusses the rapid developments in augmented reality and 
mixed reality technologies, from a practitioner’s perspective of making the aug- 
mented reality sculptural work In Darwin’s Garden. From its conception in 2012, to 
its exhibition at Carbon Meets Silicon II in 2017, the advances in augmented reality 
technology led to an interplay between the goal of the creators and the technological 
realisation of that vision. The art, design and technology involved, generated a reac- 
tive process that was mired in external influences as the accessibility to augmented 
reality became commercially valuable and subsequently restricted. This chapter will 
be of interest to anyone who wants to understand more about the possibilities, tech- 
nologies and processes involved in realising mixed reality practice and about the 
commercial culture that supports it. 


Keywords Augmented reality - Sculpture - Extended realities - Transmediation - 
Embodiment + Virtuality 


18.1 Introduction 


The artwork In Darwin’s Garden was developed by the artist Chris Meigh-Andrews 
with the collaboration and assistance of Rowan Blaik, Head gardener at Down House, 
and the author, a design educator and researcher at the University of Chester, UK. 

Chris Meigh-Andrews’ art practice considers the resonance of place, by using 
moving image to discuss temporality and the relationships to space, of history and 
natural forces. He produces site-specific installations that examine spatio-temporality 
using technological devices. In Darwin’s Garden explores the historical resonance of 
the old mulberry tree in Charles Darwin’s home, Down House in Kent. The tree was 
there when Charles Darwin lived in the house and as Chris explained in an interview 
for the Leonardo Electronic Almanac: 
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is a living link to the past, and through that connection there is a tangible nexus with the 
history of ideas, to science, and to the development of theories that have profoundly shaped 
our sense of what it is to be human [1]. 


The artwork comprises of two related works where the mulberry tree is central 
to the viewer’s experience. In the first work, the three-dimensional structure and 
environment are digital; viewed on a screen or recently explored using a virtual 
reality headset. The second work is a physical sculptural form, with an augmented 
digital object at its centre. 

The digital representation of the tree used in both works was formed from photo- 
graphic imagery that included a year of time-lapse photography collected from four 
cameras placed in the garden of Down House. This body of time-lapse photography 
was the catalyst for the development of a digital 3D form. Chris Meigh-Andrews 
discussed how “3D visualisation seemed a natural extension and progression from 
previous work, and deeply related to the fact that the mulberry tree is itself a growing, 
living organic structure” [1]. 

The mulberry tree is now in decline and requires human intervention through the 
use of man-made structures to support the weaker branches (Fig. 18.1). In Darwin’s 
Garden uses full-size facsimiles of these structures in both forms of the artwork. 
In the augmented sculptural form, these create a physical space that the digital tree 
inhabits and the viewer engages with this physical space, exploring the digital form 
within it. The collaborative challenge was how to implement this augmented reality 
sculpture, combining both digital and physical elements in order to facilitate an 
immersive experience for the viewer. 

The completely digital form of In Darwin’s Garden is a three-dimensional struc- 
ture set in an environment that can be viewed on a web browser, and more latterly this 
has been explored using a virtual reality headset. Within a web browser, this work 
is viewed as a window into a digital space, seen as another reality. The viewer can 
move around this digital space using a keyboard or mouse, but their physical reality 
is separated through the implicit knowledge that the artwork is a reality contained 
within the screen. In a virtual reality format, the viewer stands in front of the structure 
with the real world expunged from their sight. Virtual movement is by a handheld 
controller but the viewer’s body does not move due to hardware and safety restric- 
tions. The virtual reality experience is exactly that, a virtual reality that disappears 
when the headset is removed and the viewer returns to the real world. 

These web and virtual reality (VR) implementations, while effective in their own 
right, do not offer what the use of augmented reality (AR) brings to the experi- 
ence. Part intention, part necessity, the AR version of In Darwin’s Garden brings an 
embodiment to viewing the artwork that is worthy of independent discussion. The 
technical journey to this augmented realisation is also of interest as the technology’s 
rapid development posed interesting questions of remediation across realities. This 
chapter discusses AR and its use within a project that combined physical and digital 
structures, theoretical notions of digital and physical space and also details of the 
technical elements of AR for art practice. There is also the relationship between 
practitioner and technology and the evolution that occurs over time, as both evolve 
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Fig. 18.1 The mulberry tree. Copyright © C. Meigh-Andrews 2019, reproduced by permission 


while developing the project; that is an important thread running through this chapter. 
My background is in industry and education, with a design practice that researches 
the understanding of digital space and notions of place, often working with com- 
puter animation and game engines. The approach to this collaboration was from the 
aspect of both designer and researcher into the possibilities of digital realities. The 
development of AR was, and still is, resolving the design and theoretical language 
needed to discuss its application. In 2011 Papagiannis likened AR to early cinema 
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...dn its infancy, when there were as yet no conventions. AR, like cinema when it first 
emerged, commenced with a focus on the technology with little consideration to content, 
marked as secondary [2]. 


The rapid and continued development of AR means it will take time before there 
are clear artistic conventions, for as Pearson states technologies are “not developed by 
the artistic community for artistic purposes but by science and industry to serve the 
pragmatic or utilitarian needs of society” [3]. This is reflected in the artistic devel- 
opment of In Darwin’s Garden, which was at the mercy of commercially focused 
technological developments. But, as Malina states when discussing art forms enabled 
by the computer, “since contemporary culture is being driven by contemporary sci- 
ence and technology, one of the roles of the artist is as “coloniser” of the technology 
for artistic end” [4]. Chris Meigh-Andrews stated this was an extension to his previ- 
ous work—it was definitely an extension to my knowledge of designing commercial 
applications for digital realities—so with the artistic vision discussed, we set out to 
explore the possibilities for an augmented sculpture. 


18.2 Augmenting Reality 


To augment reality is to blend another reality with a person’s physical reality, so 
both realities are perceived as a single seamless environment. In the most common 
and current context the phrase “augmented reality” is thought of as blending digital 
imagery with the surrounding environment. It is often discussed as a technological 
development and not as a theory of blending a different reality with the real world. 
The most identifiable forms of augmented reality are looking at a mobile phone screen 
and seeing digital objects blended with the live camera feed, or more recently using 
smart glasses that directly overlay the digital elements onto the user’s field of view. 
Yet the theoretical concept of augmented reality relates to more than a modern digital 
development and has a broader historical context in art and design practice. There 
are studies of artworks from Hellenistic times that superimpose painted realities onto 
interior walls, in a form designed to create the optical experience of both artwork and 
real world as a single reality [5]. The scientific development of perspective during the 
Renaissance led artists to the development of trompe |’ oeil, mixing real and painted 
realities. Neither trompe l'oeil nor digital augmented realities require the viewer to 
lose their sense of reality; both blur the optical boundaries between what is real and 
what is not. 

Renaissance theories of perspective are important discussions for the future of AR 
as the Albertian perspectival system that actualised trompe |’ oeil is still driving the 
Cartesian logic of today’s computer space and subsequent digital realities. McGuirk 
and Summers argue that augmented reality technologies bring “psychological and 
even philosophical concerns with regard to those [Renaissance] perspectival systems 
that underpin these technologies” [6]. And recommend the investigation of other 
forms of perspective for AR such as ocular, floating-point or multi-perspective forms 
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that are less Western specific. Artists do not rely solely on Renaissance theories of 
perspective, so it may now be time for the artistic “colonisers” of AR technologies 
to explore other theories for developing practice and as Papagiannis discusses, not 
just remediate the current contexts and forms of AR but break out from the current 
environment AR is discussed in [2]. 


18.3 Extended Realities 


The discussion of augmented reality in its digital form must be understood in the 
context of other digital realities as technology continues to drive this area forward and 
create new forms. A catch-all term used for this area is Extended Realities, commonly 
abbreviated to XR. It refers to all forms of combined real and virtual environments 
including augmented reality (AR), virtual reality (VR) and mixed reality (MR). 
If a spectrum is considered, where a real environment and a completely virtual 
environment are at opposite ends, then the range in between these two endpoints 
is where realities are mixed in different ratios. This is Milgram’s Reality-Virtuality 
Continuum [7], a concept to aid in the creation of a taxonomy for mixed realities. 
The possibility is that this range may not be discrete but continuous and it has since 
been extended in another dimension using mediality. In order to be clear for this 
discussion, we shall briefly consider what defines virtual, augmented and mixed 
realities. 

Virtual reality is a completely immersive reality that replaces the user’s real-world 
environment with a simulated one. A headset covers the user’s eyes, headphones cover 
their ears, then visual imagery and audio immerse the user in a digital environment. 
There may also be elements of haptic feedback to enhance the immersion. 

Augmented reality, in its digital form, is where the real world is overlaid with 
computer-generated imagery with the intention to create the illusion of a single 
seamless environment. In order to avoid limiting AR to specific devices, Azuma [8] 
stated three characteristics for AR: 


1. Combines real and virtual; 
2. Interactive in real time; 
3. Registered in three dimensions. 


These were written before mobile phones became powerful enough to run AR 
technologies and so were prescient in their thinking. 

Mixed reality is where virtual objects can interact with real-world objects in a 
form where the user can act on either, or both, with them having an appropriate 
reaction. A mixed reality headset uses sensors to map the real objects and track their 
movements within the field of view so digital objects can appear attached to real 
objects. The headset can recognise hand movements and the user can manipulate a 
digital object as if it were in the real world. 
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Extended realities will continue to develop and so definitions may be redefined 
by future technological advances. 


18.4 The Drivers Behind the Development of Augmented 
Reality 


Compute-driven augmented reality was first explored in 1968 by Sutherland [9] 
but remained a specialist area until the early 2000s when increases in processing 
power in affordable mobile devices enabled AR to become a viable technology with 
which to engage users. The development of a variety of AR software development 
kits (SDK’s) for mobile devices meant developing AR applications became steadily 
more accessible to anyone who understood a certain level of coding. Companies 
released SDK’s free to developers, but this meant the developer was locked into that 
company’s technology [10]. 

An alternative to using an SDK is the use of augmented reality browsers. Devel- 
opers register with the browser company and can then create AR experiences by 
uploading digital files to the company’s cloud service. The browser will then down- 
load these files to a user’s device. This business model locks both developer and user 
to the specific AR company and their browser technology. 

The use of AR browser applications has enabled graphic designers to create printed 
material containing images that play video or show three-dimensional objects when 
the reader uses the specific browser on their phone. With this development’s ease 
of use, an array of print-based AR advertising was created as a way of attracting 
attention to products and appealing to mobile user’s curiosity to try AR experiences. 
The monetization of AR as an advertising format instigated the development of an 
array of applications and SDK’s and the developers of mobile devices took note. 

Metaio GmbH was an augmented reality development company that started in 
2003 and was used in the initial prototypes for In Darwin’s Garden. They provided 
an AR web browser and development kit for programming AR applications for com- 
puters, web and mobile devices. In May 2015, it was reported that Apple had bought 
the company [11] and Metaio announced all products and subscriptions were to be 
discontinued. Developers using Metaio had to find another way to serve augmented 
reality experiences to their users before the deadline of December 2015. In Septem- 
ber 2017 Apple released iOS 11, their mobile operating system for iPhones and 
iPads, that included support for augmented reality development. Their application 
programming interface (API), called ARKit, allows third-party developers to build 
augmented reality applications that can take advantage of the device’s functional- 
ity and processing power. The hardware developers of mobile devices were now 
directly supporting the development of augmented reality on their products through 
their operating systems. 
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18.5 Augmenting Art 


In contrast to the evolving monetization of AR, artists recognised the opportunities 
for their practice. In October 2010 Sander Veenhof and Mark Skwarek created the 
WeARinMoMaA exhibition in the Museum of Modern Art (MoMA), New York. This 
was not an official MoMA exhibition, but as the exhibition’s website states an inva- 
sion showcasing the “radical new possibilities and implications Augmented Reality 
is bringing to the cultural and creative field” [12]. Using the Layar augmented reality 
browser application [13], an Internet connection and the global positioning system 
(GPS) on their phones, visitors were able to see digital objects inserted into MoMA’s 
gallery spaces. Since then this form of unauthorised intervention has resulted in a 
range of dialogues between artists and exhibition spaces, exemplifying the conflict 
and opposition to traditional conventions both Pearson and Papagiannis discussed 
[2, 3]. MoMA is still being used in this way with the 2018 MoMAR project target- 
ing permanent displays in the museum for AR artwork interventions [14]. On the 
WeARinMoMA website, Veenhof and Skwarek added a cheeky “PS The MoMA is 
not involved yet” [12], and while MoMA has never responded they do now have at 
least one artwork in their collection that uses AR, Martine Syms, Incense Sweaters 
& Ice, 2017, [15]. 


18.6 In Darwin’s Garden: Producing the Digital 
Environment 


Production on In Darwin’s Garden started with the web-based digital environment, 
as this was the less experimental production process. It was intended that digital 
elements from this web format could then be used in the creation of the second 
format, the augmented reality sculpture. Both artworks could be split into component 
parts of the tree and the supporting structure. The tree would be digital in both, but 
the supporting structure would be digital in the web format and physical in the AR 
format. The relationship between the component parts and their true to life scale to 
the viewer was deemed crucial to the experience. 


...the old tree is now in its decline, with man-made structures supporting some of its branches. 
We would like to make full-size facsimiles of these structures to use as a foil to the virtual 
image of the tree and develop a work that would enable visitors to explore the tree in virtual 
space and time—Meigh-Andrews [1]. 


The viewer of either experience should encounter a full-size tree and supporting 
structure. This should not be a perfect digital representation of the tree in three- 
dimensional space, as this would not carry the empathic connection of standing next 
to a living, dying tree. The viewer in this experience has to feel they are up close to 
a very specific mulberry tree, physically stepping around the framework supporting 
the old tree, getting close to the trunk and looking up through the canopy to the sky. 
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At the time production started there was experimental photo software, such as 
Photosynth [16], that could crowdsource photos of a landmark and then build a digital 
photo cloud simulation of the landmark from them. This approach was a precursor to 
photogrammetry, now used in the 3D scanning of real objects, where multiple photos 
are captured and the software extracts information from these to build an exact three- 
dimensional representation. An exact replication was not the intention for Jn Darwin’s 
Garden, but a representative photographic form where the viewer builds their own 
mental image of the real tree through the photo cloud approach offered interesting 
possibilities. This format also had links to David Hockney’s two-dimensional photo 
collages—teferred to by the artist as “joiners’—but in a three-dimensional form that 
would be at the actual scale of the subject matter. 

Using 3D modelling software, the tree structure was assembled as a photo cloud 
comprising of single planar shapes with a photo applied to each. These photo planes 
were arranged in such a way as to create the abstracted form at the scale of the real 
tree. The physical framework supporting the tree was modelled so as to link with the 
tree in a manner representative of the real-world site. The time-lapse images were 
placed as clouds of photo planes around the structure in locations that were spatially 
representative of the camera locations (Fig. 18.2). 

The main technical consideration for this design approach was the quantity and 
quality of photographic images, more images equate to a larger file size and an 
increased download time. The photos included the time-lapse imagery meaning a 
single photo plane would contain 20 time-lapse photos, played in a looping sequence, 
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Fig. 18.2 The web-based form of In Darwin’s Garden. Copyright © A. Summers 2019 
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and there were 20 photo planes for each of the four cameras. In 2012 the average 
speed in the UK was around 12.0 Mbit/s so the first version of the application at 
86 Mb would take just over 1 min to download. This was considered to be too long. 

To reduce file sizes, a consideration of how close a viewer will get to a single 
image was determined. It was expected that a viewer would be able to walk up to 
and into the lower areas of the tree structure. This meant that a single image in the 
lower section of the tree could fill the viewer’s screen. Therefore, any image at this 
lower level must be the same resolution as the screen the viewer is using. A standard 
resolution, at that time, for playing a standalone application on a computer monitor 
was 1024 by 768 pixels. 

Photo planes in the middle of the tree are at a height above the viewer’s eye line so 
could be a lower resolution, as no single image will ever fill the viewer’s screen. The 
photo planes at the top of the tree were bigger in scale, but fewer in number, in order 
to create areas of foliage and blue sky. These needed to be at the screen resolution 
as these would again fill the viewer’s screen when looking upwards. 

The author’s previous design practice had explored the analysis of how trees 
move in the wind, applied to game environments, where wind and weather affected 
how a player interacts with an environment. Wind movement was explored for In 
Darwin’s Garden using an algorithm to create a random direction and strength of 
the wind. Photo planes fluttered emulating foliage in the wind and added something 
to the presence of the tree structure, but it was considered too much of a simulation; 
ironically it did not feel natural in the digital environment. On testing it was left 
running over 24 hours, only to discover an issue in the wind algorithm meant that the 
photo planes moved imperceptibly over time. The whole tree structure would move 
across the space, effectively walking out of the environment. A key development 
point was to remember to leave each iteration running for a reasonable length of 
time, as it might in a gallery situation. 

Experimenting with programming the photo planes to face the viewer meant that 
an element of movement was present but not overpowering. As the viewer moves, 
the photo planes overlap and intersect while rotating to face the viewer. This kept a 
sense of physicality and real-world movement within the abstracted foliage. 

In its web-based form, In Darwin’s Garden explores space and temporality sur- 
rounding the notion of place. The viewer can see Down House in the background 
locating them in the space of the garden. Within that space the tree’s supportive frame 
is treated as a physical barrier the viewer must move around, while the photo planes 
of the tree offer no resistance to movement and the viewer can move through them. 
The photo planes containing the time-lapse imagery hold the location of the real 
time-lapse cameras making the viewer spatially aware of the image of the garden 
space and the real-world tree. This web version was finalised and uploaded to the 
IDG web site [17] and in August 2012 presented via Leonardo Electronic Almanac’s 
digital media exhibition platform [18]. 
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18.7 Producing the Augmented Reality 


18.7.1 Production Process 


At this point in production with digital assets assembled, the consideration of how to 
transfer the conceptual vision to an augmented reality experience began. Transmedi- 
ation across forms of extended realities is not a direct process, as virtual reality offers 
complete immersion which brings greater levels of control over the viewer’s experi- 
ence than augmented reality, where the viewer can see their physical surroundings. 
The transmediation of In Darwin’s Garden was not a simple transfer of digital assets; 
it meant overcoming technical issues while working with the dialogue between digi- 
tal and physical forms, leading artist and designer on an explorative journey into the 
possibilities of augmented reality. Design enquiry alongside technical investigation 
is required for each of the component parts in an AR experience and this is a useful 
way to break down and reflect on the process. Those component parts are 


Viewing Device: this runs the application and requires a screen and camera. 
Augmented Reality Application: to be installed on the viewing device. 

Digital Asset: to be displayed on the viewing device. 

A Positioning Target: often an image but may also be a physical object or a GPS 
location that triggers and positions the display of the digital asset. 


BRUDE 


The principal for an AR application is that when launched it accesses the viewing 
device’s camera, displaying the camera feed on the screen of the device. When the 
camera is pointed at a target image the application recognises this image and displays 
the digital object superimposed onto the camera feed. The digital object holds a fixed 
spatial relationship with the target image, so when the device is moved around the 
target image, the display shows the viewer to be moving around the digital object. 


18.7.2 Viewing Device 


The ability to position the work in any indoor or outdoor location was important 
for the development of In Darwin’s Garden. In an indoor gallery, a viewing device 
can be supplied with a preinstalled application. Using a specific device means any 
application can be fully tested to ensure it works properly. Devices can be updated 
and interesting developments in technology, such as advances in augmented and 
mixed reality glasses, can be explored. 

In an outdoor context, it is unlikely that a viewing device can be left securely at 
a site, so the viewing device has to be the mobile phone the viewer is carrying. The 
application will need to be developed for a range of devices and registered with the 
appropriate application stores. Downloading the application will be limited by the 
viewer’s network transfer speeds and data allowance so its file size will be a factor in 
any viewing experience. If a person encounters the work outdoors and is not carrying 
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a suitable device, the engagement with the sculpture will only be through its physical 
form. 


18.7.3 Application Development 


To use a pre-existing augmented reality browser would mean any viewer first has to 
download it from the Internet, then the digital object is downloaded while the browser 
is running. This is potentially the fastest development process and advantageously 
uses a proven and tested browser. But it brings in the extra stage for the viewer of 
dealing with the AR browser and its brand, which can act as a barrier to the process 
of engaging with the artwork. If an application is built specifically for the work it 
will contain all the information needed and is the only item to be downloaded. The 
disadvantage is that the application will need to be developed, tested and certified in 
order to be distributed by an application store. Also instructions on where and how 
to download any application must be visible near the installation. Augmented reality 
applications and SDK’s for building applications are constantly being developed and 
improved upon but a useful starting point is A Comparative Analysis of Augmented 
Reality Frameworks Aimed at the Development of Educational Applications [19]. 

The decision was made to use the Metaio software tools to develop the augmented 
reality sculpture. This provided access to Junaio, a free mobile AR browser appli- 
cation for iOS and Android devices, along with software development kits for pro- 
gramming, PC, web and mobile augmented reality applications. The Metaio Cloud 
stored content online and the Metaio Creator software was a very good drag and drop 
creation system. Metaio also organised the insideAR conferences at the forefront of 
technological developments in this field. This suite of software gave flexibility in 
terms of choice of device, application development or a ready to use AR browser, 
and an offline or cloud-based approach. Metaio also supported the Epson Moverio 
BT-200 Smart Glasses that contain the processing power of a smartphone and the 
ability to superimpose the device screen upon the field of view of the user. These 
offered exciting possibilities for AR experiences so the development of the AR form 
of In Darwin’s Garden began using Metaio tools, smart devices and Epson’s Moverio 
glasses. 


18.7.4 The Digital Object 


There are various forms of digital object that can be displayed using AR; a two- 
dimensional graphic in vector or raster form, a video file, or a three-dimensional 
digital object which may be static or animated. Images, text and video are two- 
dimensional objects that can be mapped directly onto a target object to overlay it, or 
programmed to appear above it and always face the camera. A digital object might be 
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“smart” meaning it can continually access device functionality, such as GPS location, 
to provide a constantly updated flow of information. 

In 2013, AR advertising was starting to become more commonplace. AR was 
being used to play videos over car advertisements in magazines and to show digital 
replicas of the advertised car [20]. Three-dimensional objects can sit directly on or 
over the target object aligned to face a specific direction and also animated. A major 
factor in building digital objects to use in AR is file size. The larger the file size, 
the longer its download will be and the more likely the user will move on before 
the AR experience gets started. In the web version of In Darwin’s Garden, the size 
of the digital asset for the tree was 36 Mb; this meant a minute or more download 
time over a 3G mobile network. To explore file size options different iterations of the 
digital asset were created by adapting the number of photo planes, the resolution of 
the photos and the number of time-lapse images in each animation. The augmented 
experience required a level of transmediation led by file sizes also considering a 
viewer’s accessibility to an Internet connection, the viewing context in terms of 
location, and the active engagement with the augmented reality experience. 

Firstly, the time lapse images were relocated to be in the tree structure amongst 
the images of foliage. A physical reason for this was the viewer might not be able to 
walk across a gallery in order to see the images as in the web version. Importantly 
they were now embedded in the experience of the tree, like fruit amongst the foliage. 
The viewer could recognise that some photo planes contain a sequence of landscape 
images as opposed to images of tree foliage. These became objects that draw the 
viewer into the digital space, so that they move in the real world around and through 
the physical framework. An interesting time-lapse image is where a portrait of the 
artist appears because Chris Meigh-Andrews was working on the camera as it took 
a photo, this creates the chance that a viewer will encounter the artist within the AR 
experience. 

In the augmented experience, the backdrop photos of the house were removed 
because they show a fixed landscape behind the digital tree that is not there within 
the real-world context; this would break the viewer’s immersion. Instead the view 
only holds the digital tree and behind it the viewer’s actual location appears, situating 
this digital form in the real landscape with the support structure. This means other 
viewers can be seen as part of the view, situating them in both the digital and physical 
realities, and true to scale against the digital tree (Fig. 18.3). 


18.7.5 Target and Positioning 


The most basic problem that limits the immersiveness of an augmented reality expe- 
rience is that of registering the digital asset correctly over the target [8]. Advances in 
the technology, needed to register a digital file in the correct position and hold it there, 
have made a huge impact on user immersion in extended realities. Early AR systems 
relied solely on images, known as fiducial markers, with the developer registering 
each target image to display an appropriate digital file. The markers placed in the 
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Fig. 18.3 The augmented digital tree structure seen to scale. Copyright © A. Summers 2019 


environment were often black and white block patterns which could be distinguished 
in terms of direction to the camera and angle of view. These attributes could then be 
used to place the object on the target image with the correct spatial and perspectival 
relationship to the camera. Developments in image processing and camera resolution 
in mobile phones has meant that photographic images can now be used as targets, 
although contrast and asymmetry in a target image are important in order to calculate 
direction and angle of view. 

Another positioning system used is the device’s global positioning system (GPS). 
This is reliant on the efficiency and stability of the GPS signal and limitations occur for 
indoor locations where GPS information may be hard to determine. If GPS tracking 
is unstable then small changes in location can make the digital object appear to jump 
around in front of the viewer. 

Metaio, then subsequently Apple’s ARKit, can use an object as a target. This 
requires the AR application to know the shape of the object in order to calculate 
direction and angle of view. Developers create separate applications dedicated to 
scanning objects for use as a target. 


348 A. Summers 


Advances in environment recognition from camera feeds has resulted in the ability 
to position a digital file using markerless tracking. With this system, the application 
recognises the planar surface in the camera view and places the digital file onto this 
surface. This means there is no need to prepare an environment with markers as the 
digital file will locate itself and move around on any flat surface in front of the viewer. 

The first iterations of In Darwin’s Garden used the GPS tracking function within 
the Metaio software to align the digital tree form within the physical frame. Using the 
application outdoors, the digital structure jumped around due to issues of tracking and 
an inability to receive accurate location information. The sculpture is 5 m by 3.2 m 
and as the viewer walked around the sculpture, the GPS tracking was inconsistent. 
When the errors in tracking caused the digital tree to move only a few centimetres 
this was deemed acceptable, but when the digital tree would suddenly move a number 
of metres it immediately broke the immersion, as it appeared to jump outside of the 
physical structure. Further tests in a gallery space found that some devices might not 
be able to receive any GPS location information. After various tests, it was decided 
an image-based target was to be used as it was the most stable form of tracking. 

An issue with positioning a large digital object is the tracking of that object when 
the viewing device moves off the tracking image. As the viewing device pans up the 
tree structure, the camera loses sight of the target marker and the digital structure 
would become unstable. This is also an issue where a digital object moves out of the 
field of view but the user will expect the digital object to still be there upon returning to 
view that part of the environment. This is commonly referred to as extended tracking 
and has been important in developing the possibilities for AR for dealing with more 
than a single digital asset in the user’s environment. 

During the development of In Darwin’s Garden, Metaio developed object tracking 
and provided an application for scanning 3D objects. A problem was the large scale 
of the physical structure, as this technology was geared towards scanning small 
objects such as toys. Experiments with scanning a smaller maquette of the physical 
framework worked at the scale of the maquette but it was found that this could not 
be scaled up to match the size of the sculpture. 

A development with the Metaio software was the ability to upload a digital model 
of the object that could be used for tracking a real object. This proved successful 
in recognising the large framework, but only if the whole framework could be seen 
within the camera view. If the viewer was too close to the physical frame, then the 
tracking could not recognise a component part in order to position the digital asset. 
If the viewer could be directed to approach the sculpture from a certain direction, 
where the camera would have a full view of the structure, then this was a viable 
method of tracking. 

The experimentations in tracking eventually led to the use of a single image 
marker placed in the centre of the framework as this offered the most consistent 
stability during the developmental stages of production (Fig. 18.4). 
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Fig. 18.4 CAD render of the 
modular framework showing 
the circular image marker. 
Copyright © A. Summers 
2019 


18.8 Exhibiting In Darwin’s Garden 


In 2015 a modular framework for the physical structure was constructed. Five steel 
units were fabricated to hold a wooden framework together that can support the two 
“A” frames that lean inwards as if they are supporting the real mulberry tree. The 
structure had been modelled in CAD, (Fig. 18.4), in order to create the necessary con- 
struction diagrams and this also meant an exact digital object file could be extracted 
to be used for tracking within the Metaio software. 

In early November of 2015, the augmented reality sculpture was installed outside 
the Chester Contemporary Art Space for testing. The devices chosen to run the 
application where iPads and the Epson Moverio BT-200 Smart Glasses. Using a 
Metaio application, the iPads picked up the structures’ shape and the digital tree 
form would hold within the structure with minor stability issues. Using the Epson 
Smart glasses, the lower resolution camera feed proved to be able to track the object 
but only in a good light. When marker-based tracking was used, the tree structure 
tracked with reasonable stability on both devices. Then in December 2015 Metaio 
revoked all licences for their software, as they had reportedly been bought by Apple 
in May of that year [11]. This stopped all applications from working and there was 
a certain irony that In Darwin’s Garden, acting as a form of an archive for the dying 
mulberry tree, had itself become obsolete within 3 years of its conception. 

There followed a period of 6 months of testing other SDK’s and applications as 
commercial products competed to fill the gap in the market left by Metaio. The SDK 
from Vuforia was used to create a marker-based tracking system for In Darwin’s 
Garden, but the possibilities of object tracking were not available. 
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Fig. 18.5 Using the iPad holder to view In Darwin’s Garden. Copyright © Wrexham Glyndwr 
University 2019, with permissions 


In September 2017, In Darwin’s Garden was installed as part of the Carbon Meets 
Silicon 2 exhibition at the Oriel Sycharth Gallery, Wrexham. For this installation, the 
smart eye glasses were not used as these could not be secured at the site. Instead iPads 
were placed in specifically designed plywood holders that slot into charging stations. 
These holders were robust, having handles at both ends to allow two hands to hold 
the device (Fig. 18.5). They were designed to give confidence in handling the device 
and remove the fear of holding, or dropping, an expensive electronic device. This 
appeared to change the nature of the user interaction as users were more confident 
holding these larger objects. The users were quick to step into the physical space 
and interact with the digital elements and each other. The interconnection between 
the physicality of the real world and the nuances of the digital world appeared to be 
enhanced by giving the viewer confidence in their handling of the device required to 
engage with the digital reality. 


18.9 Conclusions 


Throughout the development of In Darwin’s Garden, there has been the need to 
react to the advancements of augmented reality technology, as well as the disap- 
pearance of that technology when it became commercially valuable. This shifting 
dialogue between technology and artwork raises interesting questions of transmedi- 
ation and archival. Do we keep developing an artwork until the technology facilitates 
the vision, or accept current technological limitations and compromise that vision? 
In Darwin’s Garden is complete, yet for each new installation it is expected that 
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the digital assets will need to be embedded in new devices because of software and 
hardware developments. 

In its gallery iteration, it became apparent that there is an embodiment which 
occurs between the viewing device and user that facilitates a more complete inter- 
action with In Darwin’s Garden. This embodiment is inherent in the physicality of 
holding the plywood iPad holders with both hands and moving around and through 
the sculpture as these devices appear to push the digital foliage out of the way. This 
may be in part due to the true to life scale of the digital element that facilitates a tacit 
understanding within body movement and the path the viewer takes exploring the 
digital tree. This embodiment between user and viewing device may also be due to the 
connection we have with our personal smart devices. We are confident in positioning 
ourselves and our smart device in order to get the best photo. Subsequently, we have 
a familiarity with the spatial connection between a camera view and targeting objects 
of interest within it. 

The smart glasses used in developing In Darwin’s Garden were found to have a 
narrower field of view than human vision, meaning a frame appeared to clip the edges 
of the digital tree. This broke viewer immersion, as the frame appeared to float at a 
constant distance away from both viewer and object. As this technology develops to 
allow wider fields of view this disconnect should reduce, but in this instance these 
devices did not facilitate a truly immersive experience for the AR sculpture. 

In the end it was the physicality of holding a framing device in a form where 
the user can twist and angle it with their body, hold it out or look in close, adapting 
the screen view to their preference, that was found to be the most intuitive form for 
the exploration of an augmented sculpture of this size. Personal smart devices can 
facilitate this intuitive exploration, so for external installations of In Darwin’s Garden 
it will be important for viewers to download the application in order to engage with 
the experience using a familiar device. 

In conclusion, there has been a constant tension between developing the aug- 
mented artwork, utilising technological advances and adapting to setbacks as access 
to technology was withdrawn. With each advance there was a temptation to be led by 
technology, but the transmediation of the vision, not the application of technology, 
was the essential driver throughout the development process. 
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Chapter 19 A) 
Creation of Interactive Virtual Reality edt 


Scenarios as a Training and Education 
Tool 


Rinat R. Nasyrov and Peter S. Excell 


Abstract A novel approach to the creation of realistic training scenarios for safety- 
critical industrial applications is presented. This is based on virtual reality techniques 
but extended by the incorporation of a range of options for interactivity, which per- 
mit the trainee to take actions in realistic ways in the simulated environment. These 
actions may include potentially dangerous errors, with realistic consequences simu- 
lated, but in complete safety. Any real environment may be simulated both visually 
and functionally in the virtual environment. An innovative feature is the use of virtual 
buttons displayed on the image of the user’s hand, thus avoiding the need for acces- 
sories such as haptic gloves. The system enables trainee specialists to gain realistic 
operational experience without the anxieties of causing damage in a real environ- 
ment, but it is also relevant to a wide range of applications where rich interactivity 
is needed. 


Keywords Interactivity - Learning - Safety-critical - Scenarios - Training - Virtual 
reality 


19.1 Introduction 


It is probable that the majority of uses of modern Virtual Reality (VR) technologies 
are for entertainment. However, VR technologies also routinely allow people working 
in safety-critical industries to undergo training in a realistic environment, without 
the risk of hazards such as injury or equipment damage. Electrical power distribution 
is an example of a major safety-critical industry and a need arose to create a virtual 
training simulator for power substation operators, who have to be ready to react 
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rapidly, correctly and safely to a wide range of routine and emergency situations. 
The work presented here discusses the development of an innovative VR training 
system for such cases, but the principles established could be used in entertainment 
or creative applications as well. 

The target need arose from an approach by the Russian power industry, for which 
over 30% of faults and blackouts have historically been caused by errors during 
switching [1] and hence the improvement of operators’ training is a priority: it is 
probable that the situations in other industries and in other industrialized countries 
will show significant similarities. 


19.2 Limitations of Current Two-Dimensional Simulators 


Many simulators exist for training of industrial personnel, but they mainly rely on 
two-dimensional screen displays. There is no doubt that the underlying algorithms are 
sophisticated; for the power substation operator case they typically include: control 
of simulated operation order with both routine and non-routine events; estimation 
and recording of the operators’ decisions made during such simulated events; rapid 
evaluation of the parameters of the modelled system. 

Against this, such simulators have deficiencies, such as: a two-dimensional dis- 
play of the main control room and substation equipment does not give a realistic 
scenario to develop skills; routine normal operations tend to be perfunctory and with- 
out deep insight for the trainees; navigation around the installation is not a realistic 
representation of the real situation. 

Given the powerful impact of three-dimensional simulations now routinely avail- 
able for entertainment applications, it was felt that the development of such tech- 
nology could deliver a major improvement in training simulators. A VR simulator 
for industrial operatives would be vastly more meaningful, although there would 
need to be significant improvements in the quality of interaction, in comparison with 
entertainment-oriented VR devices. 


19.3 Three-Dimensional Approaches to Scenario 
Simulation 


19.3.1 3D Technologies 


There are two main three-dimensional training technologies: 3D Helmet and 3D 
CAVE (cave automatic virtual environment). Both of them can be used as forms 
of VR simulator and both deliver complete immersion in the virtual environment. 
However, the CAVE demands a relatively large amount of space: it leaves the user 
unfettered by a headset, but nonetheless has significant deficiencies of realism: in 
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particular, the user only sees a 2D projected image, albeit on a surrounding wall. 
Thus, if the space is limited or true 3D is essential, the 3D Helmet technology is 
preferable. 

The 3D CAVE consists of a cube-shaped space, normally with a volume of at 
least 8 m?, with display screens surrounding the user on all sides [2]. The user may 
work with virtual objects with hands or a joystick, but it is difficult to display these 
realistically since the surrounding screens are basically two-dimensional; further, 
the user’s mobility is limited by the cube perimeter. For 3D Helmet VR technology, 
two small displays are mounted in front of the user’s eyes so that they can view the 
simulated environment [3]. The user then has the opportunity to work with credible 
3D representations of virtual objects, either with their hands or a joystick. 


19.3.2 Modelling Requirements for Appropriate 
Environments 


A primary need is a 3D model of the environment: this must have a good degree of 
verisimilitude in both its visual and spatial similarity to the real thing, but it must 
also have functional similarity in the operation of relevant switches, indicators, etc. 
The verisimilitude of a 3D model leads the user (operator) to develop the skills of 
operation of the selected environment during the training, such that after completion 
they will not need to adapt to the real environment, due to the realism of the VR 
interaction. 

Spatial analogue representations of objects and the environment are very impor- 
tant: operators need to know the time needed to move from one point to another in 
the space concerned and this can be critical in emergency scenarios. Further, the need 
for functional similarity in the 3D model is paramount and the model must react to 
the user’s actions in the same way as in the real environment. 


19.3.3 Scenarios of Training 


The scenarios of training have to include both regular and emergency cases: regular 
operations refine the skill of routine switching and understanding of the basic oper- 
ation on the station. Emergency scenarios need to start in the same way as regular 
training but then there will be the insertion of an emergency at a time and of a nature 
that must be unpredictable to the trainee, who then has to take correct decisions as 
fast as possible [4]. 

The user can choose between training mode and test mode. If they choose training 
mode, they can select either emergency or regular training and the type of scenario. In 
test mode, the user does not know what type of training scenario they will encounter: 
it will be a random choice of the simulator. 
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19.4 Implementation of Virtual Reality Simulator 


A substation of typical Russian design was chosen to be simulated. This contained: 
six 110 kV overhead line connections; six 35 kV overhead lines and one 35 kV 
cable line connection; thirty 10 kV cable line connections; outdoor switchgear for 
110 kV; outdoor switchgear for 35 kV; indoor switchgear for 10 kV; three three- 
phase transformers. The area of the prototype was about 18,000 m?. This diversity 
of equipment enables many scenarios to be implemented. 

The digitalization of the prototype has three main stages, as follows [3]: acqui- 
sition of a cloud of object points for the prototype hardware; mesh calculation and 
optimization; texture calculation and optimization. The object point cloud becomes 
a skeleton model in the virtual space: the surfaces of the prototype objects are then 
given appropriate textures and the whole 3D model is optimized. 

Creation of training scenarios is a specialized issue dependent on the applica- 
tion. For the substation operator case, the regular scenarios included taking out of, 
and putting into, service the following: transformers; circuit-breakers; disconnec- 
tors; overhead lines. For the emergency cases, the following scenarios were chosen. 
1. Taking circuit-breakers out of service due to: emergency powering off of trans- 
former; emergency voltage transformer fuse failure; emergency current transformer 
explosion. 2. Taking an overhead line out of service due to emergency collapse of 
column insulation of disconnector. 3. Taking a sulfur hexafluoride circuit-breaker 
out of service due to emergency gas depressurising. 4. Bringing a disconnector into 
service due to an emergency earth fault on an overhead line. The totality of these 
scenarios covers about 90% of typical emergency operations in a substation. 


19.5 Technical Description of the Simulator 


The game engine part of the simulator is based on Unreal Engine 4 [5] (Epic Games, 
2014). A software model of the chosen prototype was written in the C language, 
enabling the latency to be reduced to less than 10 ms, which is essential for a con- 
vincing experience. The VR headset used an umbilical cable to link to the host 
computer because there is no current wireless link technology that can transmit two 
channels of HD video plus three channels of gyroscopic information and a USB-3 
channel (for the forward-looking camera—see below) with acceptable latency. 

There are three main stages of the creation of the training system (Fig. 19.1): 
preparatory stage, software creation, system operation. 

The preparatory stage consists of the selection of the target environment, carrying 
out 3-D scanning of the real objects in that environment, creation of a library of 3-D 
models of objects and equipment in the environment, and finally documentation and 
refinement of the scenarios (Fig. 19.2). 
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Stages of simulator 
creation 


Software creation: 


System operation: 
- Documentation 
creation; 

- Maintenance; 

- Instructing on 
training procedures; 
- Commissioning. 


- Training simulator logical 
part: capabilities and 
application; 


Preparatory stage: 
- Creation of 3-D equipment 
model library; 

- Documentation and 
refinement of training 
operating scenarios. 


- 3-D shell (engine) 
capabilities/application; 
- Integration of the 3-D shell 
into the training simulator; 
calibration of 
interworking interface. 


Further development 
without implementer’s 
participation 


Further development 
involving implementer’s 
participation 


Fig. 19.1 Creation stages of VR training system 


To create a functioning simulator, itis necessary to combine the technical functions 
of the modelled system (the simulator logical part) with the capabilities of modern 
3-D shells (graphic engines). The integration of these two subsystems is discussed 
in Sect. 19.6.1. 


19.6 Software Implementations 


19.6.1 Functional Simulations and Graphics Engine 


In principle, the system consists of two subsystems that constantly interact and 
exchange information. The first subsystem is the logical part of the simulator: this 
performs the technical algorithm that describes the real behaviour of the simulated 
system and its response to operator actions, including button pressing, switch knob 
turning, cable unplugging and, above all, mistakes. 

The second subsystem is a graphical 3-D shell (graphics engine) responsible for 
the visual presentation of the substation model. The connection between the two 
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Environment selection 
3-D scanning 
Environment modelling 


3-D model library 
for equipment/ 
objects 


Elements of 3-D 
modeling and their 
interoperability 
with the simulator 
logical elements 


Writing a software 
model of the scenario 
for the simulator 
logical part 


Simulated model testing 


Simulator commissioned into 
operation on the basis of the 
environment model 


Fig. 19.2 Component preparation for the developed system 


subsystems is event-based and is performed through the interaction interface. The 
protocol is thus that the simulator logical part proceeds to a new state calculation 
for the system only after receiving the appropriate command from the 3-D shell, this 
resulting from the user having performed some action that triggers the need for a 
new response. Following this, the simulator logical part evaluates the action taken 
and, if it causes a linked visual change in the environment display, sends a command 
to the 3-D shell to display the corresponding change. 

The main training station acts as a server, to perform functions of administration, 
programming and information storage. The trainee interacts with the 3-D shell using 
a specific set of controllers connected to their personal computer. There are three 
options for the sets of controllers, with increasing levels of functionality: 


1. Minimum: the user can effect changes in the simulated equipment using a min- 
imum set of peripheral devices—a monitor, keyboard and mouse are needed, 
connected to a computer. 

2. The optimal set of controllers consists of a virtual reality helmet with gyroscopic 
position sensing and a forward-looking camera to track the position of the user’s 
head, arms and body. 
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3. Maximum: differs from the optimal by adding an omnidirectional treadmill, 
which allows the trainee to move around in a virtual environment, ensuring the 
maximum presence effect, and consequently the effectiveness of training. 


19.6.2 Operating Modes 


The system can operate in two basic modes: functional preparation mode and train- 
ing mode. The functional preparation mode consists of environment preparation 
(see Fig. 19.2): the equipment models are created, connection diagrams loaded and 
training scenarios are recorded in the corresponding databases on the server. 

The training mode is configured for two types of system operation: a multiplayer 
mode and an individual training mode. A schematic diagram of the operation in 
multiplayer training mode is shown in Fig. 19.3. In this mode, the personal computers 
of the staff in training are connected to a single local area network which includes 
the main training station, which is an automated workstation for the instructor. There 
are servers for the 3-D shell and simulator logical part at the main training station, 
providing communication between training stations during group training sessions, as 
well as a complete database covering training scenarios and the library of equipment 
in the environment. Administration and programming functions are possible from 
the main training station. Programming involves entering training scenarios into 
the simulator logical part using the “Form editor” application, as well as further 
operational adjustments as required by the scenarios created. 


The trainee 
The main Training Training Training 
training station station No1 station No2 station Ne..N 
Hardware Hardware Hardware Hardware 
3-D shell 3-D shell 3-D shell 3-D shell 
<—_____> << — > << 
server Required substation Required substation Required substation 
scene and scene and scene and 
equipment modules equipment modules equipment modules 
library for training library for training library for training 
Simulator Simulator Simulator Simulator 
logical part lg > logical part < > logical part < > logical part 
server Substation wiring Substation wiring Substation wiring 
diagrams and diagrams and diagrams and 
training scenarios training scenarios training scenarios 


Fig. 19.3 The training mode system structure, when the main training station is a separate supervisor 
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Basic individual training can be conducted on a dedicated PC: the simulator 
application is installed at the trainee’s workplace; a model file of the trainee’s or 
instructor’s choice is then loaded. Depending on the configuration, model files can 
be located on a workstation or on a server with access to model files using the CIFS 
protocol. 


19.7 Creation of Scenarios 


Analogous to a computer game, the graphics engine can create an arbitrary environ- 
ment into which the trainee can enter when using the VR headset. However, as stated 
above, to reduce latency to less than 10 ms, as is essential for a realistic experience, 
the use of high-level language and packages was avoided and instead the model was 
entirely written in the C language. For the prototype that was developed, the client 
required a realistic model of a high-voltage substation, both the outdoor high-voltage 
equipment and the indoor control room and low-voltage equipment. All of this had 
to have simulated functionality, including moving parts, to cover all of the requested 
training scenarios. 

To create the environment model simulating a real example of a substation, about 
1200 working hours were required, occupying six people, a modeller (mesh creator), 
two programmers to create mathematical models, one unifying programmer and two 
electrical power engineers. The three-dimensional information was derived partly 
from scale drawings and photographs and partly from three-dimensional laser scans. 
The results show an excellent degree of verisimilitude (Fig. 19.4). Further images 
showing the realism that was achieved in modelling of the outdoor hardware are 
given in Figs. 19.5, 19.6 and 19.7. 

The same techniques were used in modelling of the interior of the control room 
and low-voltage switch room (Figs. 19.8 and 19.9), although much more functional 
detail had to be included here, such that switches could be turned, buttons could be 
pressed, lights could come on and even plugs on flexible cables could be pulled out 
in the virtual environment. Figure 19.10 shows the detail that was incorporated for 
this purpose: the plugs shown may be pulled out of their sockets in the virtual world 
and all of the switches, buttons, and lights function in it as well. 

For the full VR experience, the equipment shown in Figs. 19.11 and 19.12 is 
required. The volume of the region that can be accessed is determined by the placing 
of the motion sensors attached to the tripods, but a cube with 3 m side length is 
usually adequate, especially if an omnidirectional VR treadmill is available [6]: in 
fact, the treadmill would allow a smaller volume to be used, e.g. a cube with side 
length of 2 m. 

Figure 19.13 shows the front view of the headset, which has a forward-looking 
infrared camera attached to its front: this is used to collect the imagery of the user’s 
hand so that the system can determine what the hand is doing and it can also be used 
to implement some virtual controls on the hand (see below). Figure 19.14 shows the 
pair of stereoscopic views that are generated in the two eyepieces of the headset, 
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Fig. 19.4 Top: photograph of the real substation that was simulated. Lower: modelled equivalent 
(differences are mainly due to a slightly different viewing angle) 
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OK 


Fig. 19.5 3-D model of 110 kV outdoor switchgear yard 


Fig. 19.6 3-D model of a 110 kV sulfur hexafluoride circuit-breaker 


including a representation of the user’s hand reaching to press a button. Figure 
shows the system in use: the user is wearing the VR headset and reaching out to 
grasp one of the plugs attached to a flexible cable (Fig. ). The left image shows 
the view that the user experiences: the virtual image of the hand is shown extracting 
the plug from its socket in the switchgear. 

Figure is a diagrammatic recreation of the “virtual buttons” that are made 
to appear on the left hand of the user in the virtual view. The cube below the wrist 
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Fig. 19.7 3-D model of a 110 kV disconnector 


Fig. 19.8 3-D model of the main control room 


is touched when it is desired to change the view significantly, for instance, to move 
from the control room to the outside switchyard in the combined representation of the 
substation. The virtual button near the left-hand thumb is used to advance the user’s 
virtual standing position forward in a way rather similar to Google Streetview. The 
virtual button in the palm of the left hand has been provided for future functionality 
but is not currently used. 
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Fig. 19.9 3-D model of 10 kV indoor switchgear group 


as 


Fig. 19.10 Detailed 3-D functional model of 10 kV circuit-breaker 


There are no virtual controls on the right hand, but the system detects the index 
finger pointing forward as a tool to press buttons, while a grasping action using all 
of the fingers is seen as a tool intended to turn a control knob or grasp a plug. 
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Fig. 19.11 The appearance of the simulator’s core elements 


3 meters Construction 
$ > height 3 meters 
~ 


3 meters 


E 


<— Ea 


Fig. 19.12 Plan layout of the simulator’s elements 
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Fig. 19.13 Trainee operator interacting with a 3-D model: the forward-looking camera is noticeable. 
The marker on the index finger was optional in an early version, but not now needed 


19.8 Outcomes and Conclusions 


A three-dimensional stereoscopic software representation of an industrial installation 
was created, including a large range of virtual interactive functions. 

The implementation was designed to exploit a virtual reality headset with 
enhanced functionality, including gyroscopes to determine the movement of the head- 
set and a forward-looking infrared camera which was used to detect the position of 
the hands and fingers of the user, thus avoiding the need for haptic gloves. Further, 
virtual buttons were located on the hands of the user in the virtual reality space, thus 
avoiding the need for hardware control buttons. The implementation also prioritized 
extremely low latency in the display, to give maximum reality in the simulation. To 
ensure this, the headset was connected to the controlling computer by an umbilical 
cable, as no standard commercial wireless link would have sufficient capacity and 
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Fig. 19.14 The synthesized stereoscopic view seen by the user shown in Fig. 


Fig. 19.15 A user interacting with the system. Right: the user is wearing the VR headset and 
reaching out to grasp one of the plugs in Fig. 19.10. Left: the view that the user experiences: the 
hand is shown extracting the plug from its socket 
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Press this button to move Press the cube with right 
forward to the point on the index finger to select an 
floor marked with a white spot area to view on the plan 


Do not use this button 


Fig. 19.16 The virtual button controls provided on the image of the left hand, as detected by the 
forward-looking camera 


adequately low latency. In addition, the code representing the simulated environ- 
ment was written in C to avoid latency issues that could be caused by higher level 
programming approaches. 

The simulated environment in the prototype was an industrial plant (electricity 
substation), created with great attention to verisimilitude. Excellent realism was 
achieved and this included functional button switches and electrical connectors, all 
of which could be operated on by the user’s own hands without the need for haptic 
gloves or other technical adaptations. The industrial environment simulated was 
dictated by the primary sponsor, but an arbitrary range of alternative environments 
can be envisaged. A potential disadvantage is the large amount of specialist labour 
needed to create the simulation: 1200 person-hours in the present case, not including 
work on assembling the hardware. 

The system functioned completely according to specifications and has been 
adopted for the training of substation operators by a major electrical power distribu- 
tion company in Russia. A particular advantage of a virtual reality training simulator 
in an application of this type is the avoidance of any danger resulting from errors 
made by the operators during their training. The system has been tested with a num- 
ber of trainee operators and results have been successful, in the sense of perceived 
realism and relevance of the training. Work is now proceeding to implement other 
environments. 
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Abstract Discipline boundaries have been systematically developed and established 
over the last century which reflect some of the perceived differences between the sci- 
ence and engineering disciplines and those in the arts and humanities. However, 
both internal and external factors can result in new disciplines which can arise at the 
boundaries between existing disciplines, rather than within them, and may combine 
different aspects of these disciplines. Some of these may be produced by exter- 
nal drivers such as information technology, the Internet and Big Data. It has been 
argued that innovative and creative approaches that cross disciplinary boundaries 
are more effective in producing new information and new knowledge. The current 
scope and priority of interdisciplinary research can be obtained by analyzing the 
funding programmes of national and international bodies and agencies. Some fund- 
ing programmes are looking for quicker and more effective solutions for complex 
grand challenge problems. Analyses of complex information systems may require 
the crossing of departmental boundaries in order to generate new knowledge. It is 
recognized that obstacles to effective and sustained interdisciplinary collaborations 
have the potential to arise unexpectedly. Also, social and cultural factors often come 
into play when new initiatives are proposed. Mechanisms to grow interdisciplinary 
research such as internal funding, networking, and training are considered. Potential 
barriers to progress are also noted. Ongoing issues that remain to be addressed are 
summarized. 
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20.1 Introduction 


In 1959 Snow identified differences between the science and engineering disciplines 
and those in the arts and humanities [1]. This appeared to be due to a variety of factors 
including tradition, vocabulary, ways of working, and contributions to society, all of 
which could be different in different disciplines and circumstances. 

Critchley [2] proposed that Snow had diagnosed the emergence of two cultures 
because of the loss of a common framework of understanding. Scientists and engi- 
neers favoured advancement of society through technology and industry, whereas 
the arts and humanities preferred intellectual and literary endeavour. 

However, Gould took an opposing point of view and emphasized the commonali- 
ties between science and the humanities [3]. In 1963, Snow appeared to take a more 
optimistic view about the relationship between science and arts [4]. 

These views stem from an era when 5% or less of the population participated in 
higher education, and a faculty member’s role was not subject to detailed perfor- 
mance management controls by their institution. The freedom in the academy pro- 
vided ample opportunity for new ideas to be developed and explored. Research grants 
were relatively easy to obtain (there were fewer applicants), and there was opportu- 
nity for regular publication of research results in refereed journals or monographs. 
Discipline boundaries had been fairly systematically developed and established over 
the previous century and these were generally respected. In addition, there was ample 
new knowledge to be discovered within the existing discipline domains. 

This chapter provides a landscape view of interdisciplinary research and devel- 
opment and examines the areas where future progress may be made. The ways in 
which the creative arts relate to this view are detailed in Sect. 20.6 of this chapter. 


20.2 Development of New Disciplines 


However, in the last half-century new disciplines have arisen and there has been an 
acceleration of the pace of change. For example, computer science developed princi- 
pally out of mathematics in the UK because of the associated interests in numerical 
analysis and the requirement to use automated methods to perform the required cal- 
culations. In the USA, computer science developed principally out of engineering 
due to the interests in hardware and manufacturing computing devices. However, 
before they emerged as separate academic disciplines with their own departments 
and faculty members, they did exist along the boundary of the existing disciplines 
of mathematics and engineering, respectively. The primary driver for the develop- 
ment of computer science came from outside the academy in the form of the vacuum 
tube computer (1940) [5], the transistor (1948) [6], and the silicon chip (1961) [7]. 
National governments appreciated the strategic significance of the technology not 
just for scientific and business computation but also for their application to a wide 
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variety of disciplines and problems, and so invested significantly in computing facil- 
ities for the academy and national research laboratories. Shockley, Bardeen, and 
Brattain were jointly awarded the Nobel Prize in physics in 1956 [8] for the develop- 
ment of the transistor. Therefore, in its initial stages, computing was seen as part of 
the natural sciences, at least in its hardware aspects. This is still the case with regard 
to quantum computing, and publications from this area can appear in the journal 
Nature [9], whereas those from other areas of computer science do not. 

Therefore, new disciplines can arise at the boundaries between existing disci- 
plines, rather than within them, and may combine different aspects of these dis- 
ciplines. Oceanography is a further example of an interdisciplinary discipline as 
it covers all aspects with respect to the ocean, which includes its natural compo- 
nents (seawater, living creatures, and the sea bed); its currents and movements (fluid 
dynamics); and modelling of the overall system (such as by finite element analy- 
sis). Similarly, cognitive science draws on interdisciplinary aspects of psychology, 
linguistics, philosophy, interaction, and computer analysis. 

Shneiderman [10] details how innovative and creative approaches that cross disci- 
plinary boundaries are more effective in producing new knowledge. In particular, he 
advances the case for combining applied and basic research and a new paradigm for 
interdisciplinary collaboration that puts engineering and design on an equal footing 
with basic science. Thus even within well-established disciplines there are significant 
opportunities for new approaches. 

Pressures on funding of universities in recent years have resulted in competi- 
tion between departments and between institutions. There is competition between 
departments for students and competition between institutions for research funding 
and league table positions. It also produced competition between disciplines that 
resulted in science and engineering arguing that they were better for students and 
society because they imparted skills and generated jobs, whilst arts and humanities 
subjects were less useful in these respects. In some cases, it resulted in national 
bodies advising students against studying arts subjects, which in turn forced some 
of these academic departments to close because their institutions could no longer 
afford them if they could not pay their way. Senior academics argued strongly for the 
value of arts and humanities [11, 12]. However, this overall antipathy between arts 
and sciences which has been somewhat artificially created has in some cases acted 
against furthering the important collaborations that are needed. Therefore obstacles 
to effective and sustained interdisciplinary collaborations have the potential to arise 
unexpectedly. The participants need to have the perspectives and resources to be able 
to deal with them. 

Further information on the advantages and disadvantages of interdisciplinary 
research are detailed in [13]. 
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20.3 Acceleration of the Pace of Change 


Friedman [14] identifies three principal components that are accelerating change 


(i) Market 
(ii) Nature 
Gii) Technology 


The market includes the effects of globalization and political tribalism. Nature 
includes climate change. Technology includes the Internet and the increasing power 
and reducing costs of computational facilities. All of these act as powerful forces 
and they act in parallel. Their combined effect can cause severe disruption. Society 
needs to be able to adapt and respond to the changes that these forces represent. Many 
disciplines and applications use the Internet as a delivery service, to access Cloud 
storage, a computational resource, or a marketing medium, and therefore become 
subject to its accelerating effects. One Internet year is said to be equivalent to seven 
calendar years [15]. One effect of the Internet has been to break down traditional 
barriers. Formerly research teams were co-located with their departments, but today 
the Internet can be used to facilitate collaboration across departments and countries 
with equal facility. This can result in a greater sharing of ideas and results. Therefore 
the potential for interdisciplinary collaboration is substantial, and the Internet can 
act as an external driver of change in the same way that the silicon chip inaugurated 
the computer revolution. 

Many institutions and organizations are increasingly recognizing the need to ana- 
lyze large data sets. This can arise in the context of experiments, sensors, simula- 
tions, and the user data gathered by social sciences and social media companies. 
In the academy, such data is often concentrated in a centralized Data Center, with 
associated expertise on hand to advise on data collection and data analysis. They 
are often resourced centrally by the academy as they are regarded as an investment 
to support all disciplines. Such central resources also offer graduate courses (e.g., 
Masters, Ph.D.) in the various areas of Big Data and its applications. Thus, Data 
Centers can have the direct effect of encouraging and facilitating cross-disciplinary 
collaborations. Therefore, the rise of Big Data, rather like the rise of the Internet, has 
become a significant internal and external driver which forces researchers involved 
in this area towards interdisciplinary collaborations. 

It is clear that interdisciplinary working is not just a matter of the right structures 
and appropriate facilitation. Social and cultural factors also come into play [16]. 
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20.4 Interdisciplinary Research Funding and Research 
Priorities 


An indication of the scope and priority of interdisciplinary research (IDR) can be 
obtained by analyzing the funding programmes of national and international bodies 
and agencies. Included in this analysis are the National Science Foundation (NSF) 
in the USA, the European Union, and the UK funding agencies. 


20.4.1 National Science Foundation (NSF) 


NSF states the priority it gives to interdisciplinary research in terms of advancing 
scientific discovery and extending the fields of knowledge [17—19]. It also supports 
interdisciplinary training in order to enable the recipients to be able to address current 
problems in innovative ways [17, 18]. 

The following challenges were identified [18]: 


Collaboration across people with different backgrounds and cultures 

Extra time needed to build consensus and learning new methods 

Traditions and policies tend to allocate resources to traditional areas 
Professional societies could assist more by active support of interdisciplinarity 
Ensure the peer review process including interdisciplinary researchers. 


Potential lessons from industry and national laboratories were as follows [18]: 


e Industrial and national laboratories are generally organized to address the 
problems 

e Management is more top-down than in the academy 

e There is potential for greater collaboration between the academy and industry. 


20.4.2 European Union 


A report to the European Union (EU) on interdisciplinary research in the context of 
ERA and Horizon 2020 identified a number of reasons why this should be a priority. 
This included the following [20]: 


Faster solutions are needed for grand challenge problems 
Engage in problem-driven approaches 
Exploit disruptive innovation 


e 
e 
e 
e Bridge the gap between research communities. 
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The report also outlined the potential effects of interdisciplinary research [20]: 


Trigger innovation 

Add value to the disciplines 

Solve complex real-life problems 

Bring about discoveries outside traditional disciplines 
Change established research perspectives and paradigms 
Achieve impact. 


The following enablers were identified for interdisciplinary research and devel- 
opment [20]: 


Resourcing of people, time, and space 

Providing targeted funding 

Developing interdisciplinary skills and practices 

Triggering of appropriate social and cognitive dynamics between researchers 
Ensuring undergraduate students receive a grounding in different disciplines 
Facilitating an understanding of different scientific vernaculars. 


The EU has also explored the reasons for the support of interdisciplinary research 
in disruptive technologies [21]. 


20.4.3 UK 


Research England was set up as a Council of UK Research and Innovation by the 
Higher Education and Research Act 2017 [22]. It exists and alongside the other 
existing Councils (the seven Research Councils [23] and Innovate UK [24]). One of 
its functions is to evaluate the extent to which interdisciplinary research and devel- 
opment. It includes including cross-disciplinary, multi-disciplinary, interdisciplinary 
and transdisciplinary activity. 

A number of reports on interdisciplinary research (IDR) have been undertaken by 
various bodies in the UK. Each body has particular interests to do with their mission, 
so each report is oriented to their objectives. However, taken together the reports 


e highlight a range of barriers and incentives for interdisciplinary research (IDR) 
in the UK 

e explore the challenges associated with assessing interdisciplinary research 

e provide some real-world examples of ‘what works’ in supporting an IDR culture 
in today’s higher education research institutions [25]. 


These are now summarized. 
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20.4.3.1 The British Academy 


The Executive Summary of ‘Crossing paths: Interdisciplinary institutions, careers, 
education and applications’ by the British Academy noted the following [26] 


e Requests for evidence identified a broad support of interdisciplinary research 
e Essential for addressing complex problems and global social challenges 
e Enhances understanding of the separate disciplines. 


20.4.3.2 Global Research Council 


A Survey Report for the Global Research Council (GRC) in 2016 noted the role and 
importance of IDR, the right conditions for establishing interdisciplinary working; 
assessment, evaluation, and measurement; careers, training, and recognition; and 
produced a number of recommendations including the following [27]: 


Better sharing of best practice 

The identification of grand challenge problems 

The provision of funding support over adequate time frames 
The provision of physical and social space 

Fair review processes 

Appropriate end of project evaluation metrics 

Training and capacity building 

Improving awareness of the value and importance of IDR. 


20.4.3.3 Review of the UK’s Interdisciplinary Research 


A review of the UK’s interdisciplinary research using a citation-based approach was 
undertaken for HEFCE and MRC by Elsevier. In order to assess interdisciplinarity, 
the diversity of article bibliographies was studied in eight comparator countries for 
the period 2009-13. The principal conclusions were that interdisciplinary activity 
was growing in the UK; it has an international collaboration component, and the 
academy was a principal contributor [28]. It also noted a lower citation impact for 
interdisciplinary research. 


20.4.3.4 Landscape Review 


A landscape review of interdisciplinary research in the UK identified some potential 
areas where new stakeholders could promote suitable conditions for interdisciplinary 
research to develop [29]. 
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20.4.3.5 Case Study Review of English Higher Education Institutions 
(HEIs) 


A Case Study review by Technopolis identified a number of ways of organizing IDR 
such as [29] 


© Co-location of researchers 

e Researcher networks across subject areas, departments or faculties 

è Researcher-led (‘bottom-up’) and/or strategic institutional (‘top-down’) 
approaches 

e A thematic or generic focus for IDR 

è Support for high-quality research in general, not specifically for IDR. 


It also suggested ways of growing IDR such as seed corn funding, network- 
ing events, improving the skills base by appropriate Masters courses and Ph.D. 
programmes [29]. 


20.4.3.6 Team Science in Biomedical Research 


A report to the Academy of Medical Sciences in 2016 summarized the characteristics 
of team science including benefits and challenges, reward and recognition, and career 
progression [30]. 

An assessment of progress was done in 2019 [31]. 


20.4.3.7 Arts and Humanities Research Council 


The Global Challenges Research Fund (GCRF) is a 5-year £1.5bn fund and a key 
component in the delivery of the UK Aid Strategy: tackling global challenges in the 
national interest. The funding was for 1 December 2018 onwards [32]. The objective 
includes interdisciplinary research excellence. 


20.5 Interdisciplinary Research and Development 
in the Creative Arts 


20.5.1 Introduction 


The continued increase of processing power, data storage, telecommunications band- 
width, and display screen resolution provide greater opportunities for handling new 
types of users and applications. It also increases the ubiquity and facility of compu- 
tational devices in terms of mobility and user interfaces, whether utilized directly by 
designers, audiences, users, or embedded in the application environment. 
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Art and design have a long history in antiquity. They have shaped the values, 
social structures, communications, and the culture of communities and civilizations. 
The direct involvement of artists and designers with their creative works has left a 
legacy enabling subsequent generations to understand more about their skills, their 
motivations, and their relationship to the wider world, and to see it from a variety 
of perspectives. This in turn causes the viewers of their works to reflect upon their 
meaning for today and the lasting value and implications of what has been created. 

Some historical examples of art and design were able to use semi-automated 
methods for creation, particularly where large areas of a canvas or model needed 
to be filled in. However, it was only with the advent of modern technology that the 
advantages of harnessing digital techniques were able to be exploited. One of the 
earliest examples was the Architecture Machine [33] designed and implemented at 
the Massachusetts Institute of Technology (MIT). The objective was to enable digital 
technology to assist the user with design tasks, particularly those at large and small 
scales, where it was known that designers had particular challenges and difficulties. In 
addition, the computer was able to store data and reproduce designs, thus facilitating 
the speed-up of the iterative process towards a final design which met the objectives 
of the designer and the requirements of the client [34]. It also enabled the aesthetics of 
the design to be seen at the design stage within the wider content of the environment 
in which it was to be placed. This was subsequently extended using virtual reality 
technology to enable prospective users of the building to perform ‘walk-throughs’ 
within the created space and understand how it would work in practice in terms of 
logistics, to ensure it was fit for the purpose. Materials and implementation costs 
could also be optimized at the design stage. 

More recent examples of artists and designers interacting with technology include 
the use of the iPad to produce sketches of scenes which were subsequently grouped 
into a montage to give a large wall display containing multiple images [35]. Mobile 
phones with high-resolution screens are being used to produce art works and com- 
municate them via social media networks [36]. Art installations have also harnessed 
modern technology both to process information and to display it. Such environ- 
ments have proved useful in engaging users and visitors with real-time images and 
interactive art. 

Collaborative design has enabled the sharing of information across digital net- 
works to produce designed objects in virtual spaces. Augmented and virtual reality 
techniques can be used to preview designs before they are finalized and implemented. 
Ancient and modern art and design environments illustrate the design and implemen- 
tation processes involved, and the opportunities for collaborating and interacting with 
other artists and designers. 

There are also increasing opportunities for artists to use their skills and exper- 
tise to illuminate the latest discoveries in science and technology [37, 38]. There is 
increasing interest in extracting meaning from very large data sets, which in turn 
requires effective methods of analysis and presentation of the results. Science and 
technology are also able to contribute increasingly to the arts, either as a compo- 
nent of the artistic output, or part of the methodology used to produce the output. 
In addition, the traditional boundaries between arts and technology are becoming 
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blurred due to the way computing technology is being embedded into the everyday 
environment in a seamless way, and the use of social media which enables a greater 
degree of involvement and sharing by the community. Social media can open up new 
dimensions of interaction and participation in both the arts and the sciences [34]. 


20.5.2 Contributions of Technology to the Creative Arts 


Technology contributes to the creative arts by providing tools, interfaces, facilities 
for collaboration, access to data, as well as a range of software for a variety of 
applications. Such facilities may also include artificial intelligence and machine 
learning techniques. Telecommunication networks enable artists to participate in 
communities and collaborate with other artists on joint projects. Exhibitions can 
be displayed on the Internet and allow local and remote audiences to participate 
and interact with the project and each other. Such environments can also be used 
for a variety of applications including simulation, planning, theatre rehearsal, and 
entertainment, such as computer games [39—42]. Excellent artistic skills are important 
for the generation of the scenes for effective gameplay. 

Artistic works may use digital methods to produce digital art [43]. Computer 
games may be considered as a particular kind of digital art as they use computer 
technology and associated software to produce the game. 


20.5.3 Contributions of the Creative Arts to Technology 


The creative arts have made contributions to Renaissance Teams, the design of tech- 
nology, and the effective incorporation of technology into the wider environment. 
Artists have been able to contribute to scientific analysis and enable new results to 
be produced [44, 45]. Such Renaissance Teams can stimulate creativity and produce 
new knowledge [46] in way that would be very difficult, if not impossible, for a single 
artist to do on their own. SciArt is a term used to describe the artistic contributions 
that can be made to scientific investigations [47]. 

Technology companies give increasing attention to the aesthetics, usability, and 
engagement of the devices that they create (e.g., Apple [48, 49]). 

An example of the analysis of cultural artefacts by computer is the Digital 
Michelangelo Project at Stanford University [50, 51]. Its objectives were to advance 
the technology of 3D scanning and to create a long term digital archive of significant 
cultural artefacts in Italy. These archives were made generally available over the 
Internet. 
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20.6 Methodologies for Identification and Assessment 
of Research 


A number of methods to evaluate research are being used by institutions and national 
bodies. These include journal quality (such as by impact factor and internationality), 
paper citation rates, impact factors, H-indices of the individuals and departments, and 
altmetrics. The UK Research Excellence Framework (REF) also has international 
representation on its evaluation panels in order to provide externality, and to seek to 
align internal review processes with externally accepted standards. 

A number of difficulties and challenges arise in seeking to extend these methods 
of assessment within single disciplines to research delivery across a number of dis- 
ciplines. Identification and assessment of interdisciplinary research are summarized 
in a consultancy report [52], which noted that different metrics can produce different 
results [53]. 


20.7 Conclusions 


20.7.1 Ongoing Issues 


Although interdisciplinary research has been identified as a priority by NSF, EU, and 
the UK, and resources have been identified to give grant and infrastructure support, 
the following ongoing issues remain to be fully addressed 


(1) Strong discipline silos are still the norm in higher education and graduate 
research and there seems to be little inclination to change apart from a rela- 
tively small number of institutions such as Arizona State University [54-56] 
and Pohang University of Science and Technology (Postech) [57]. 

(2) External examiners who adjudicate on Ph.D. theses would not normally have 
interdisciplinary experience and expertise sufficient to give an appropriate and 
fair evaluation of an interdisciplinary thesis. 

(3) Graduate students can feel that an interdisciplinary project may be too risky, 
and also may have an uncertain outcome. 

(4) Budget structures in higher education are often ring-fenced around specific 
disciplines. It can also be difficult to allocate part of the budget across an internal 
boundary. 

(5) Interdisciplinary research appears to be mainly stimulated by external drivers 
such as information requirements or technology developments. These drivers 
often require new ways of working. This results in interdisciplinary research 
operating in reactive mode rather than seeking to advance proactively according 
to a detailed agenda. 
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(6) Moving away from a traditional discipline into a new area can still be perceived 
as risky due to the inertia in the academy, the difficulty of applying research 
metrics to new areas, and uncertainty about the future of the new discipline. 

(7) Increasing competition for grant funding and funding within institutions, can 
force review bodies to be more conservative and only allocate funding to well- 
established areas, when such funding is limited. 


20.7.2 Enablers for Collaboration Between the Arts 
and the Sciences 


Traversing the boundaries between disciplines can be a challenge. However, recent 
developments have brought about a number of enablers. Firstly, the Internet has pro- 
vided a greater ease of communication across disciplines and between researchers. 
It has also flattened organizational hierarchies to some degree, which in turn has 
reduced the traditional barriers between existing disciplines. Secondly, the model 
of Renaissance Teams [58] has proved useful in bringing together interdisciplinary 
groups of researchers with complementary expertise sets and defined goals. Thirdly, 
there have been significant successes in the application of information technology to 
the arts and humanities (for example, in the digitization of ancient manuscripts and 
artefacts). Fourthly, art and design have been successfully applied in support of the 
creative processes involved in generating new technology products and services, and 
in the presentation of scientific results. Fifthly, the increasing amounts of data being 
generated by research projects have resulted in the establishment of Data Science 
Centers within the academy, with teams of faculty and researchers brought together 
from various disciplines. Sixthly, grant funding bodies are increasingly focussing on 
larger initiatives often traversing different disciplines and countries (e.g., European 
Union grants) so require personnel to work together on advanced research and devel- 
opment. In addition, creative and artistic cooperation has demonstrated the benefits 
for all parties. All these factors make interdisciplinarity more the norm for the future 
and can generate a level of momentum and validity which provide reassurance and 
support for new researchers who wish to enter the area. 
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